Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:08:50.414940 6472 test_util.cc:268] Using random seed: 111015229
I20240501 14:08:50.421767 6472 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:50.421877 6472 ts_itest-base.cc:113] --------------
I20240501 14:08:50.421911 6472 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:08:50.421934 6472 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:50.421949 6472 ts_itest-base.cc:116] --------------
2024-05-01T14:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:50Z Disabled control of system clock
I20240501 14:08:50.430940 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:36231
--webserver_interface=127.6.82.62
--webserver_port=0
--builtin_ntp_servers=127.6.82.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.82.62:36231
--enable_range_replica_placement=false with env {}
W20240501 14:08:50.532658 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.532862 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.532898 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.535830 6484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:50.535880 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.535899 6484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:50.535915 6484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:50.539389 6484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.82.62:36231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:36231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.82.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.540187 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.541178 6484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.548115 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.548152 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.548188 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.548328 6484 server_base.cc:1034] running on GCE node
I20240501 14:08:50.548804 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.549685 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.550781 6484 hybrid_clock.cc:648] HybridClock initialized: now 1714572530550767 us; error 27 us; skew 500 ppm
I20240501 14:08:50.556293 6484 webserver.cc:460] Webserver started at http://127.6.82.62:37161/ using document root <none> and password file <none>
I20240501 14:08:50.556872 6484 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.556938 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.557121 6484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.558781 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "40a06b59079646809c1ea078c4941b5c"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.559237 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "40a06b59079646809c1ea078c4941b5c"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.562439 6484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240501 14:08:50.564602 6498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.565599 6484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:50.565721 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
uuid: "40a06b59079646809c1ea078c4941b5c"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.565800 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.579778 6484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.580348 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.580494 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.586516 6484 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.62:36231
I20240501 14:08:50.586524 6550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.62:36231 every 8 connection(s)
I20240501 14:08:50.587436 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:50.590564 6551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:50.595110 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c: Bootstrap starting.
I20240501 14:08:50.596226 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 6484
I20240501 14:08:50.596366 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:50.597608 6551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:50.598461 6551 log.cc:824] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:50.600358 6551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c: No bootstrap required, opened a new log
I20240501 14:08:50.602932 6551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a06b59079646809c1ea078c4941b5c" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 36231 } }
I20240501 14:08:50.603102 6551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:50.603137 6551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a06b59079646809c1ea078c4941b5c, State: Initialized, Role: FOLLOWER
I20240501 14:08:50.603538 6551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a06b59079646809c1ea078c4941b5c" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 36231 } }
I20240501 14:08:50.603657 6551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:50.603758 6551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:50.603859 6551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:50.604522 6551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a06b59079646809c1ea078c4941b5c" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 36231 } }
I20240501 14:08:50.604804 6551 leader_election.cc:302] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a06b59079646809c1ea078c4941b5c; no voters:
I20240501 14:08:50.605036 6551 leader_election.cc:288] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:50.605327 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:50.605573 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [term 1 LEADER]: Becoming Leader. State: Replica: 40a06b59079646809c1ea078c4941b5c, State: Running, Role: LEADER
I20240501 14:08:50.605886 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a06b59079646809c1ea078c4941b5c" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 36231 } }
I20240501 14:08:50.606122 6551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:50.607331 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a06b59079646809c1ea078c4941b5c. Latest consensus state: current_term: 1 leader_uuid: "40a06b59079646809c1ea078c4941b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a06b59079646809c1ea078c4941b5c" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 36231 } } }
I20240501 14:08:50.607460 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.607803 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40a06b59079646809c1ea078c4941b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a06b59079646809c1ea078c4941b5c" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 36231 } } }
I20240501 14:08:50.607859 6565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:50.607890 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:50.611557 6565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:50.616250 6565 catalog_manager.cc:1347] Generated new cluster ID: 59fb83743a5649cfb0dd0559616519a4
I20240501 14:08:50.616303 6565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:50.625167 6565 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:50.625999 6565 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:50.640884 6565 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 40a06b59079646809c1ea078c4941b5c: Generated new TSK 0
I20240501 14:08:50.641549 6565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:50.649714 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.1:0
--local_ip_for_outbound_sockets=127.6.82.1
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--builtin_ntp_servers=127.6.82.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 14:08:50.770946 6575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.771148 6575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.771194 6575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.774143 6575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.774242 6575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.1
I20240501 14:08:50.777734 6575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.1
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.778606 6575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.779708 6575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.787422 6580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.787423 6581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.787748 6575 server_base.cc:1034] running on GCE node
W20240501 14:08:50.787423 6583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.788198 6575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.788854 6575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.790043 6575 hybrid_clock.cc:648] HybridClock initialized: now 1714572530790023 us; error 35 us; skew 500 ppm
I20240501 14:08:50.791728 6575 webserver.cc:460] Webserver started at http://127.6.82.1:37143/ using document root <none> and password file <none>
I20240501 14:08:50.792236 6575 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.792292 6575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.792470 6575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.794039 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cffb940d250449449b4a63be0c915ebb"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.794490 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cffb940d250449449b4a63be0c915ebb"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.797580 6575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:50.799650 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.800549 6575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:50.800659 6575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cffb940d250449449b4a63be0c915ebb"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.800738 6575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.830253 6575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.830938 6575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.831092 6575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.831611 6575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.832487 6575 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.832533 6575 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.832571 6575 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.832603 6575 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.840750 6575 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.1:43531
I20240501 14:08:50.840780 6702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.1:43531 every 8 connection(s)
I20240501 14:08:50.841593 6575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:50.846015 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 6575
I20240501 14:08:50.846103 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:50.848295 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.2:0
--local_ip_for_outbound_sockets=127.6.82.2
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--builtin_ntp_servers=127.6.82.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:50.849915 6703 heartbeater.cc:347] Connected to a master server at 127.6.82.62:36231
I20240501 14:08:50.850111 6703 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:50.850596 6703 heartbeater.cc:511] Master 127.6.82.62:36231 requested a full tablet report, sending...
I20240501 14:08:50.851887 6515 ts_manager.cc:194] Registered new tserver with Master: cffb940d250449449b4a63be0c915ebb (127.6.82.1:43531)
I20240501 14:08:50.852967 6515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.1:51021
W20240501 14:08:50.951222 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:50.951515 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:50.951596 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:50.954521 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:50.954661 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.2
I20240501 14:08:50.958266 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.2
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:50.959169 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:50.960224 6706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:50.967725 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.967732 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:50.967737 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:50.967902 6706 server_base.cc:1034] running on GCE node
I20240501 14:08:50.968295 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:50.968804 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:50.970019 6706 hybrid_clock.cc:648] HybridClock initialized: now 1714572530970001 us; error 28 us; skew 500 ppm
I20240501 14:08:50.971858 6706 webserver.cc:460] Webserver started at http://127.6.82.2:36811/ using document root <none> and password file <none>
I20240501 14:08:50.972385 6706 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:50.972440 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:50.972620 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:50.974262 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b8ed235b418049ffacd7f37a484acc67"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.974758 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b8ed235b418049ffacd7f37a484acc67"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.978036 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:08:50.980091 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.981025 6706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:50.981137 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b8ed235b418049ffacd7f37a484acc67"
format_stamp: "Formatted at 2024-05-01 14:08:50 on dist-test-slave-crgf"
I20240501 14:08:50.981217 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:50.997092 6706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:50.997679 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:50.997828 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:50.998366 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:50.999290 6706 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:50.999339 6706 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:50.999398 6706 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:50.999425 6706 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.008082 6706 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.2:42581
I20240501 14:08:51.008129 6834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.2:42581 every 8 connection(s)
I20240501 14:08:51.008984 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:51.013818 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 6706
I20240501 14:08:51.013927 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:51.015990 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.3:0
--local_ip_for_outbound_sockets=127.6.82.3
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--builtin_ntp_servers=127.6.82.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:51.017575 6835 heartbeater.cc:347] Connected to a master server at 127.6.82.62:36231
I20240501 14:08:51.017776 6835 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.018272 6835 heartbeater.cc:511] Master 127.6.82.62:36231 requested a full tablet report, sending...
I20240501 14:08:51.019217 6515 ts_manager.cc:194] Registered new tserver with Master: b8ed235b418049ffacd7f37a484acc67 (127.6.82.2:42581)
I20240501 14:08:51.019758 6515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.2:34755
W20240501 14:08:51.122174 6838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:51.122371 6838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:51.122416 6838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:51.125363 6838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:51.125483 6838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.3
I20240501 14:08:51.128844 6838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.3
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:51.129673 6838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:51.130721 6838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:51.137957 6844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.137948 6843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.137960 6846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.138137 6838 server_base.cc:1034] running on GCE node
I20240501 14:08:51.138595 6838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:51.139117 6838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:51.140277 6838 hybrid_clock.cc:648] HybridClock initialized: now 1714572531140264 us; error 27 us; skew 500 ppm
I20240501 14:08:51.141829 6838 webserver.cc:460] Webserver started at http://127.6.82.3:38549/ using document root <none> and password file <none>
I20240501 14:08:51.142303 6838 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:51.142357 6838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:51.142496 6838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:51.144044 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8a9627041a2b4e41999ebccdf8b85fe7"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.144474 6838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8a9627041a2b4e41999ebccdf8b85fe7"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.147610 6838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:51.149531 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.150434 6838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:51.150527 6838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8a9627041a2b4e41999ebccdf8b85fe7"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.150592 6838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.167590 6838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.168143 6838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.168269 6838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.168807 6838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.169703 6838 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.169750 6838 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.169807 6838 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.169832 6838 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.177804 6838 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.3:43253
I20240501 14:08:51.177829 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.3:43253 every 8 connection(s)
I20240501 14:08:51.178647 6838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:51.182196 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 6838
I20240501 14:08:51.182276 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:51.184140 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.4:0
--local_ip_for_outbound_sockets=127.6.82.4
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--builtin_ntp_servers=127.6.82.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 14:08:51.186982 6966 heartbeater.cc:347] Connected to a master server at 127.6.82.62:36231
I20240501 14:08:51.187165 6966 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.187654 6966 heartbeater.cc:511] Master 127.6.82.62:36231 requested a full tablet report, sending...
I20240501 14:08:51.188563 6515 ts_manager.cc:194] Registered new tserver with Master: 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3:43253)
I20240501 14:08:51.189095 6515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.3:40683
W20240501 14:08:51.288585 6969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:51.288789 6969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:51.288853 6969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:51.291826 6969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:51.291975 6969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.4
I20240501 14:08:51.295379 6969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.4
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:51.296211 6969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:51.297281 6969 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:51.304785 6974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.304831 6975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.304955 6969 server_base.cc:1034] running on GCE node
W20240501 14:08:51.304797 6977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.305400 6969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:51.305908 6969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:51.307086 6969 hybrid_clock.cc:648] HybridClock initialized: now 1714572531307070 us; error 28 us; skew 500 ppm
I20240501 14:08:51.308787 6969 webserver.cc:460] Webserver started at http://127.6.82.4:38533/ using document root <none> and password file <none>
I20240501 14:08:51.309331 6969 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:51.309388 6969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:51.309574 6969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:51.311443 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "32b1bc4cab924910b82fcb4f49cfc046"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.311990 6969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "32b1bc4cab924910b82fcb4f49cfc046"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.315083 6969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:51.316987 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.317867 6969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:51.317962 6969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "32b1bc4cab924910b82fcb4f49cfc046"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.318037 6969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.349156 6969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.349784 6969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.349937 6969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.350450 6969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.351323 6969 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.351368 6969 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.351424 6969 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.351454 6969 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.360126 6969 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.4:41313
I20240501 14:08:51.360163 7096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.4:41313 every 8 connection(s)
I20240501 14:08:51.361056 6969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:51.369220 7097 heartbeater.cc:347] Connected to a master server at 127.6.82.62:36231
I20240501 14:08:51.369431 7097 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.369944 7097 heartbeater.cc:511] Master 127.6.82.62:36231 requested a full tablet report, sending...
I20240501 14:08:51.370239 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 6969
I20240501 14:08:51.370338 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:51.371002 6515 ts_manager.cc:194] Registered new tserver with Master: 32b1bc4cab924910b82fcb4f49cfc046 (127.6.82.4:41313)
I20240501 14:08:51.371574 6515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.4:34155
I20240501 14:08:51.372442 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.5:0
--local_ip_for_outbound_sockets=127.6.82.5
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--builtin_ntp_servers=127.6.82.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 14:08:51.474926 7100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:51.475152 7100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:51.475193 7100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:51.478040 7100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:51.478137 7100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.5
I20240501 14:08:51.481760 7100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:36231
--never_fsync=true
--heap_profile_path=/tmp/kudu.7100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.5
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:51.482558 7100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:51.483637 7100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:51.490996 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.490993 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:51.491194 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:51.491082 7100 server_base.cc:1034] running on GCE node
I20240501 14:08:51.491554 7100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:51.492025 7100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:51.493243 7100 hybrid_clock.cc:648] HybridClock initialized: now 1714572531493223 us; error 35 us; skew 500 ppm
I20240501 14:08:51.494977 7100 webserver.cc:460] Webserver started at http://127.6.82.5:39001/ using document root <none> and password file <none>
I20240501 14:08:51.495473 7100 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:51.495533 7100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:51.495724 7100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:51.497282 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b0630a708534475abfc8b7d8280cdc26"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.497736 7100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b0630a708534475abfc8b7d8280cdc26"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.500913 7100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 14:08:51.502956 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.503886 7100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:08:51.503983 7100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b0630a708534475abfc8b7d8280cdc26"
format_stamp: "Formatted at 2024-05-01 14:08:51 on dist-test-slave-crgf"
I20240501 14:08:51.504056 7100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:51.521523 7100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:51.522123 7100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:51.522267 7100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:51.522780 7100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:51.523593 7100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:51.523639 7100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.523692 7100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:51.523718 7100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:51.531759 7100 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.5:45909
I20240501 14:08:51.531782 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.5:45909 every 8 connection(s)
I20240501 14:08:51.532584 7100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:51.537477 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 7100
I20240501 14:08:51.537585 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:51.540740 7228 heartbeater.cc:347] Connected to a master server at 127.6.82.62:36231
I20240501 14:08:51.540957 7228 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:51.541429 7228 heartbeater.cc:511] Master 127.6.82.62:36231 requested a full tablet report, sending...
I20240501 14:08:51.542232 6515 ts_manager.cc:194] Registered new tserver with Master: b0630a708534475abfc8b7d8280cdc26 (127.6.82.5:45909)
I20240501 14:08:51.542762 6515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.5:46641
I20240501 14:08:51.551929 6472 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:51.562840 6515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42668:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:08:51.579685 6637 tablet_service.cc:1467] Processing CreateTablet for tablet 68e5290005df4afea91dfc0c73bcf793 (DEFAULT_TABLE table=TestTable [id=455b9646d453415bbe0ae23e77a80a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:51.579770 7031 tablet_service.cc:1467] Processing CreateTablet for tablet 68e5290005df4afea91dfc0c73bcf793 (DEFAULT_TABLE table=TestTable [id=455b9646d453415bbe0ae23e77a80a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:51.580531 6637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5290005df4afea91dfc0c73bcf793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:51.580636 7031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5290005df4afea91dfc0c73bcf793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:51.580675 7162 tablet_service.cc:1467] Processing CreateTablet for tablet 68e5290005df4afea91dfc0c73bcf793 (DEFAULT_TABLE table=TestTable [id=455b9646d453415bbe0ae23e77a80a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:51.581388 7162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5290005df4afea91dfc0c73bcf793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:51.585039 7245 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Bootstrap starting.
I20240501 14:08:51.586323 7246 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Bootstrap starting.
I20240501 14:08:51.587522 7245 tablet_bootstrap.cc:654] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:51.588218 7246 tablet_bootstrap.cc:654] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:51.588433 7245 log.cc:824] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:51.588532 7247 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: Bootstrap starting.
I20240501 14:08:51.588946 7246 log.cc:824] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:51.590329 7246 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: No bootstrap required, opened a new log
I20240501 14:08:51.590329 7245 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: No bootstrap required, opened a new log
I20240501 14:08:51.590389 7247 tablet_bootstrap.cc:654] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:51.590499 7245 ts_tablet_manager.cc:1367] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:08:51.590499 7246 ts_tablet_manager.cc:1367] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:51.591097 7247 log.cc:824] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:51.592902 7247 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: No bootstrap required, opened a new log
I20240501 14:08:51.593066 7247 ts_tablet_manager.cc:1367] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:08:51.593766 7245 raft_consensus.cc:348] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.593966 7245 raft_consensus.cc:374] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:51.594012 7245 raft_consensus.cc:729] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffb940d250449449b4a63be0c915ebb, State: Initialized, Role: FOLLOWER
I20240501 14:08:51.594271 7246 raft_consensus.cc:348] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.594431 7246 raft_consensus.cc:374] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:51.594473 7246 raft_consensus.cc:729] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b1bc4cab924910b82fcb4f49cfc046, State: Initialized, Role: FOLLOWER
I20240501 14:08:51.594431 7245 consensus_queue.cc:260] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.594867 7246 consensus_queue.cc:260] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.595685 7245 ts_tablet_manager.cc:1398] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:51.595685 7246 ts_tablet_manager.cc:1398] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:08:51.596043 6703 heartbeater.cc:503] Master 127.6.82.62:36231 was elected leader, sending a full tablet report...
I20240501 14:08:51.596066 7097 heartbeater.cc:503] Master 127.6.82.62:36231 was elected leader, sending a full tablet report...
I20240501 14:08:51.597201 7247 raft_consensus.cc:348] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.597383 7247 raft_consensus.cc:374] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:51.597430 7247 raft_consensus.cc:729] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0630a708534475abfc8b7d8280cdc26, State: Initialized, Role: FOLLOWER
I20240501 14:08:51.597846 7247 consensus_queue.cc:260] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.598692 7247 ts_tablet_manager.cc:1398] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:08:51.598706 7228 heartbeater.cc:503] Master 127.6.82.62:36231 was elected leader, sending a full tablet report...
W20240501 14:08:51.611979 7098 tablet.cc:2369] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:51.676060 7251 raft_consensus.cc:482] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:51.676244 7251 raft_consensus.cc:504] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.677280 7251 leader_election.cc:288] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0630a708534475abfc8b7d8280cdc26 (127.6.82.5:45909), 32b1bc4cab924910b82fcb4f49cfc046 (127.6.82.4:41313)
I20240501 14:08:51.678421 7253 raft_consensus.cc:482] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:51.678617 7253 raft_consensus.cc:504] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.679553 7253 leader_election.cc:288] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32b1bc4cab924910b82fcb4f49cfc046 (127.6.82.4:41313), cffb940d250449449b4a63be0c915ebb (127.6.82.1:43531)
I20240501 14:08:51.682113 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "cffb940d250449449b4a63be0c915ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b1bc4cab924910b82fcb4f49cfc046" is_pre_election: true
I20240501 14:08:51.682113 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "cffb940d250449449b4a63be0c915ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0630a708534475abfc8b7d8280cdc26" is_pre_election: true
I20240501 14:08:51.682309 7182 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cffb940d250449449b4a63be0c915ebb in term 0.
I20240501 14:08:51.682356 7051 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cffb940d250449449b4a63be0c915ebb in term 0.
I20240501 14:08:51.682773 6590 leader_election.cc:302] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0630a708534475abfc8b7d8280cdc26, cffb940d250449449b4a63be0c915ebb; no voters:
I20240501 14:08:51.683048 7251 raft_consensus.cc:2781] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:51.683128 7251 raft_consensus.cc:482] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:51.683171 7251 raft_consensus.cc:3037] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:51.684260 6657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "b0630a708534475abfc8b7d8280cdc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffb940d250449449b4a63be0c915ebb" is_pre_election: true
I20240501 14:08:51.684312 7251 raft_consensus.cc:504] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.684535 6657 raft_consensus.cc:2379] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b0630a708534475abfc8b7d8280cdc26 in current term 1: Already voted for candidate cffb940d250449449b4a63be0c915ebb in this term.
I20240501 14:08:51.684671 7251 leader_election.cc:288] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [CANDIDATE]: Term 1 election: Requested vote from peers b0630a708534475abfc8b7d8280cdc26 (127.6.82.5:45909), 32b1bc4cab924910b82fcb4f49cfc046 (127.6.82.4:41313)
I20240501 14:08:51.685012 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "cffb940d250449449b4a63be0c915ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0630a708534475abfc8b7d8280cdc26"
I20240501 14:08:51.685055 7051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "b0630a708534475abfc8b7d8280cdc26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b1bc4cab924910b82fcb4f49cfc046" is_pre_election: true
I20240501 14:08:51.685125 7182 raft_consensus.cc:3037] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:51.685163 7051 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0630a708534475abfc8b7d8280cdc26 in term 0.
I20240501 14:08:51.685165 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "cffb940d250449449b4a63be0c915ebb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b1bc4cab924910b82fcb4f49cfc046"
I20240501 14:08:51.685288 7050 raft_consensus.cc:3037] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:51.685453 7118 leader_election.cc:302] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32b1bc4cab924910b82fcb4f49cfc046, b0630a708534475abfc8b7d8280cdc26; no voters: cffb940d250449449b4a63be0c915ebb
I20240501 14:08:51.686120 7182 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cffb940d250449449b4a63be0c915ebb in term 1.
I20240501 14:08:51.686220 7050 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cffb940d250449449b4a63be0c915ebb in term 1.
I20240501 14:08:51.686237 7253 raft_consensus.cc:2741] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 14:08:51.686487 6593 leader_election.cc:302] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b1bc4cab924910b82fcb4f49cfc046, cffb940d250449449b4a63be0c915ebb; no voters:
I20240501 14:08:51.686648 7251 raft_consensus.cc:2781] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:51.686859 7251 raft_consensus.cc:686] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Becoming Leader. State: Replica: cffb940d250449449b4a63be0c915ebb, State: Running, Role: LEADER
I20240501 14:08:51.687136 7251 consensus_queue.cc:237] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } }
I20240501 14:08:51.689882 6515 catalog_manager.cc:5484] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb reported cstate change: term changed from 0 to 1, leader changed from <none> to cffb940d250449449b4a63be0c915ebb (127.6.82.1). New cstate: current_term: 1 leader_uuid: "cffb940d250449449b4a63be0c915ebb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } health_report { overall_health: HEALTHY } } }
I20240501 14:08:51.702962 6472 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:51.704481 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cffb940d250449449b4a63be0c915ebb to finish bootstrapping
I20240501 14:08:51.709599 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b8ed235b418049ffacd7f37a484acc67 to finish bootstrapping
I20240501 14:08:51.714454 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8a9627041a2b4e41999ebccdf8b85fe7 to finish bootstrapping
I20240501 14:08:51.719143 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 32b1bc4cab924910b82fcb4f49cfc046 to finish bootstrapping
I20240501 14:08:51.723296 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b0630a708534475abfc8b7d8280cdc26 to finish bootstrapping
W20240501 14:08:51.783416 7229 tablet.cc:2369] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:51.842869 6704 tablet.cc:2369] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:51.904106 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:51.905076 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:51.906149 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:51.907438 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:51.907620 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:52.021531 6835 heartbeater.cc:503] Master 127.6.82.62:36231 was elected leader, sending a full tablet report...
I20240501 14:08:52.124581 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.124581 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.124677 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.125432 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.139110 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:52.190868 6966 heartbeater.cc:503] Master 127.6.82.62:36231 was elected leader, sending a full tablet report...
I20240501 14:08:52.242348 7326 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:52.247684 7331 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:08:52.346341 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.346771 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.347589 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.348559 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.351661 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:52.563598 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.563598 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.563863 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.569710 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.576012 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:52.656019 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.656482 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.657419 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.659615 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:52.667234 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.763464 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.763636 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.764843 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.766122 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:52.784152 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:52.935932 6515 ts_manager.cc:292] Set tserver state for cffb940d250449449b4a63be0c915ebb to MAINTENANCE_MODE
I20240501 14:08:53.116938 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.117195 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.117810 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.118613 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:53.119339 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.203635 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.204043 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.204427 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.206248 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.206676 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:53.302363 7031 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.303861 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.303900 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.305505 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:53.312973 6637 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:53.340358 6657 consensus_queue.cc:237] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } }
I20240501 14:08:53.341601 7050 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Refusing update from remote peer cffb940d250449449b4a63be0c915ebb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:53.341847 7182 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Refusing update from remote peer cffb940d250449449b4a63be0c915ebb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:08:53.342085 7586 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:53.342273 7579 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:08:53.344235 7589 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } }
I20240501 14:08:53.344465 7050 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } }
I20240501 14:08:53.344537 7182 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } }
I20240501 14:08:53.347163 6513 catalog_manager.cc:5484] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "cffb940d250449449b4a63be0c915ebb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240501 14:08:53.350560 6657 consensus_queue.cc:237] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } }
I20240501 14:08:53.351785 7182 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Refusing update from remote peer cffb940d250449449b4a63be0c915ebb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:08:53.352048 7050 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Refusing update from remote peer cffb940d250449449b4a63be0c915ebb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:08:53.352150 7589 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:08:53.353000 7589 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:08:53.354269 7579 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } } }
I20240501 14:08:53.354851 7182 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } } }
I20240501 14:08:53.355039 7050 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } } }
I20240501 14:08:53.356388 6502 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 68e5290005df4afea91dfc0c73bcf793 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 14:08:53.359393 6592 consensus_peers.cc:485] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb -> Peer 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3:43253): Couldn't send request to peer 8a9627041a2b4e41999ebccdf8b85fe7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68e5290005df4afea91dfc0c73bcf793. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:53.360380 6513 catalog_manager.cc:5484] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 reported cstate change: config changed from index 2 to 3, NON_VOTER 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "cffb940d250449449b4a63be0c915ebb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } } }
I20240501 14:08:53.370561 6657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793"
dest_uuid: "cffb940d250449449b4a63be0c915ebb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50100
I20240501 14:08:53.370826 6657 raft_consensus.cc:595] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Received request to transfer leadership
I20240501 14:08:53.831521 7589 raft_consensus.cc:979] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: : Instructing follower 32b1bc4cab924910b82fcb4f49cfc046 to start an election
I20240501 14:08:53.831660 7589 raft_consensus.cc:1067] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Signalling peer 32b1bc4cab924910b82fcb4f49cfc046 to start an election
I20240501 14:08:53.832222 7050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793"
dest_uuid: "32b1bc4cab924910b82fcb4f49cfc046"
from {username='slave'} at 127.6.82.1:45379
I20240501 14:08:53.832398 7050 raft_consensus.cc:482] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:08:53.832451 7050 raft_consensus.cc:3037] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:53.833257 7050 raft_consensus.cc:504] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } }
I20240501 14:08:53.834237 7050 leader_election.cc:288] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [CANDIDATE]: Term 2 election: Requested vote from peers b0630a708534475abfc8b7d8280cdc26 (127.6.82.5:45909), cffb940d250449449b4a63be0c915ebb (127.6.82.1:43531)
I20240501 14:08:53.837956 6657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "32b1bc4cab924910b82fcb4f49cfc046" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cffb940d250449449b4a63be0c915ebb"
I20240501 14:08:53.838001 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e5290005df4afea91dfc0c73bcf793" candidate_uuid: "32b1bc4cab924910b82fcb4f49cfc046" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b0630a708534475abfc8b7d8280cdc26"
I20240501 14:08:53.838079 6657 raft_consensus.cc:3032] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:08:53.838104 7182 raft_consensus.cc:3037] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:53.838132 6657 raft_consensus.cc:729] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cffb940d250449449b4a63be0c915ebb, State: Running, Role: LEADER
I20240501 14:08:53.838335 6657 consensus_queue.cc:260] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } }
I20240501 14:08:53.838531 6657 raft_consensus.cc:3037] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:08:53.838982 7182 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b1bc4cab924910b82fcb4f49cfc046 in term 2.
I20240501 14:08:53.839304 6984 leader_election.cc:302] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b1bc4cab924910b82fcb4f49cfc046, b0630a708534475abfc8b7d8280cdc26; no voters:
I20240501 14:08:53.839365 6657 raft_consensus.cc:2454] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b1bc4cab924910b82fcb4f49cfc046 in term 2.
I20240501 14:08:53.839604 7591 raft_consensus.cc:2781] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:08:53.839785 7591 raft_consensus.cc:686] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 2 LEADER]: Becoming Leader. State: Replica: 32b1bc4cab924910b82fcb4f49cfc046, State: Running, Role: LEADER
I20240501 14:08:53.840075 7591 consensus_queue.cc:237] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } }
I20240501 14:08:53.842669 6513 catalog_manager.cc:5484] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 reported cstate change: term changed from 1 to 2, leader changed from cffb940d250449449b4a63be0c915ebb (127.6.82.1) to 32b1bc4cab924910b82fcb4f49cfc046 (127.6.82.4). New cstate: current_term: 2 leader_uuid: "32b1bc4cab924910b82fcb4f49cfc046" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:54.273277 6657 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 2 FOLLOWER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:08:54.273870 7591 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:08:54.277804 7182 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 2 FOLLOWER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:08:54.278344 7629 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:08:54.281515 6986 consensus_peers.cc:485] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 -> Peer 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3:43253): Couldn't send request to peer 8a9627041a2b4e41999ebccdf8b85fe7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68e5290005df4afea91dfc0c73bcf793. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:08:54.805711 7645 ts_tablet_manager.cc:902] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Initiating tablet copy from peer 32b1bc4cab924910b82fcb4f49cfc046 (127.6.82.4:41313)
I20240501 14:08:54.806641 7645 tablet_copy_client.cc:308] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:41313
I20240501 14:08:54.812197 7071 tablet_copy_service.cc:140] P 32b1bc4cab924910b82fcb4f49cfc046: Received BeginTabletCopySession request for tablet 68e5290005df4afea91dfc0c73bcf793 from peer 8a9627041a2b4e41999ebccdf8b85fe7 ({username='slave'} at 127.6.82.3:37299)
I20240501 14:08:54.812345 7071 tablet_copy_service.cc:161] P 32b1bc4cab924910b82fcb4f49cfc046: Beginning new tablet copy session on tablet 68e5290005df4afea91dfc0c73bcf793 from peer 8a9627041a2b4e41999ebccdf8b85fe7 at {username='slave'} at 127.6.82.3:37299: session id = 8a9627041a2b4e41999ebccdf8b85fe7-68e5290005df4afea91dfc0c73bcf793
I20240501 14:08:54.813875 7071 tablet_copy_source_session.cc:205] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:08:54.818133 7645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e5290005df4afea91dfc0c73bcf793. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:54.823269 7645 tablet_copy_client.cc:783] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: tablet copy: Starting download of 0 data blocks...
I20240501 14:08:54.823472 7645 tablet_copy_client.cc:647] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:08:54.825321 7645 tablet_copy_client.cc:523] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:08:54.856917 7645 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Bootstrap starting.
I20240501 14:08:54.861918 7645 log.cc:824] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:54.864504 7645 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:08:54.864773 7645 tablet_bootstrap.cc:492] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Bootstrap complete.
I20240501 14:08:54.865042 7645 ts_tablet_manager.cc:1367] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:08:54.867616 7645 raft_consensus.cc:348] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } }
I20240501 14:08:54.867810 7645 raft_consensus.cc:729] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8a9627041a2b4e41999ebccdf8b85fe7, State: Initialized, Role: LEARNER
I20240501 14:08:54.868119 7645 consensus_queue.cc:260] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: true } }
I20240501 14:08:54.868981 7645 ts_tablet_manager.cc:1398] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240501 14:08:54.869814 7071 tablet_copy_service.cc:342] P 32b1bc4cab924910b82fcb4f49cfc046: Request end of tablet copy session 8a9627041a2b4e41999ebccdf8b85fe7-68e5290005df4afea91dfc0c73bcf793 received from {username='slave'} at 127.6.82.3:37299
I20240501 14:08:54.869920 7071 tablet_copy_service.cc:433] P 32b1bc4cab924910b82fcb4f49cfc046: ending tablet copy session 8a9627041a2b4e41999ebccdf8b85fe7-68e5290005df4afea91dfc0c73bcf793 on tablet 68e5290005df4afea91dfc0c73bcf793 with peer 8a9627041a2b4e41999ebccdf8b85fe7
I20240501 14:08:55.252207 6920 raft_consensus.cc:1203] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 LEARNER]: Deduplicated request from leader. Original: 1.3->[2.4-2.4] Dedup: 2.4->[]
I20240501 14:08:55.844911 7681 raft_consensus.cc:1050] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046: attempting to promote NON_VOTER 8a9627041a2b4e41999ebccdf8b85fe7 to VOTER
I20240501 14:08:55.845638 7681 consensus_queue.cc:237] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } }
I20240501 14:08:55.847003 6920 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 LEARNER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:08:55.847095 7182 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 2 FOLLOWER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:08:55.847128 6657 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 2 FOLLOWER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:08:55.847623 7681 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:08:55.848040 7684 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:08:55.848820 7670 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:08:55.851356 7684 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.851948 6920 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.852389 7182 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.870710 6657 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.871838 6515 catalog_manager.cc:5484] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 reported cstate change: config changed from index 3 to 5, 8a9627041a2b4e41999ebccdf8b85fe7 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "32b1bc4cab924910b82fcb4f49cfc046" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "cffb940d250449449b4a63be0c915ebb" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43531 } attrs { replace: true } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.875751 7050 consensus_queue.cc:237] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } }
I20240501 14:08:55.876806 6920 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 FOLLOWER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:08:55.876844 7182 raft_consensus.cc:1261] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 2 FOLLOWER]: Refusing update from remote peer 32b1bc4cab924910b82fcb4f49cfc046: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:08:55.877144 7684 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:08:55.877455 7670 consensus_queue.cc:1035] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:08:55.878682 7684 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER cffb940d250449449b4a63be0c915ebb (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.879138 7182 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P b0630a708534475abfc8b7d8280cdc26 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER cffb940d250449449b4a63be0c915ebb (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.881048 6920 raft_consensus.cc:2932] T 68e5290005df4afea91dfc0c73bcf793 P 8a9627041a2b4e41999ebccdf8b85fe7 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, VOTER cffb940d250449449b4a63be0c915ebb (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } } }
I20240501 14:08:55.963785 6502 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68e5290005df4afea91dfc0c73bcf793 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:08:55.964705 6512 catalog_manager.cc:5484] T 68e5290005df4afea91dfc0c73bcf793 P 32b1bc4cab924910b82fcb4f49cfc046 reported cstate change: config changed from index 5 to 6, VOTER cffb940d250449449b4a63be0c915ebb (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "32b1bc4cab924910b82fcb4f49cfc046" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0630a708534475abfc8b7d8280cdc26" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 45909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32b1bc4cab924910b82fcb4f49cfc046" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 41313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9627041a2b4e41999ebccdf8b85fe7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 43253 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:08:55.967378 6637 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e5290005df4afea91dfc0c73bcf793 with delete_type TABLET_DATA_TOMBSTONED (TS cffb940d250449449b4a63be0c915ebb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49958
I20240501 14:08:55.967939 7700 tablet_replica.cc:330] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: stopping tablet replica
I20240501 14:08:55.968302 7700 raft_consensus.cc:2229] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:08:55.968583 7700 raft_consensus.cc:2258] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:08:55.969620 7700 ts_tablet_manager.cc:1875] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:08:56.065402 7700 ts_tablet_manager.cc:1888] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 14:08:56.065538 7700 log.cc:1196] T 68e5290005df4afea91dfc0c73bcf793 P cffb940d250449449b4a63be0c915ebb: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/68e5290005df4afea91dfc0c73bcf793
I20240501 14:08:56.066339 6502 catalog_manager.cc:4830] TS cffb940d250449449b4a63be0c915ebb (127.6.82.1:43531): tablet 68e5290005df4afea91dfc0c73bcf793 (table TestTable [id=455b9646d453415bbe0ae23e77a80a4d]) successfully deleted
I20240501 14:08:56.321372 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.321504 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.325590 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.332932 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.333098 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.404914 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.407693 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.409305 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.415652 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.421306 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.486163 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.486510 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.488337 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.489128 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.492256 6769 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.529247 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 6706
I20240501 14:08:56.715006 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.716351 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.716398 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.716763 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.789186 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.789884 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.791998 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.800279 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.908126 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.908356 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.908933 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.909703 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.989199 6900 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.989818 7031 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:08:56.990105 7162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:56.992549 6637 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:57.024811 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 6575
I20240501 14:08:57.031025 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 6838
I20240501 14:08:57.037050 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 6969
I20240501 14:08:57.043315 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 7100
I20240501 14:08:57.049610 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 6484
2024-05-01T14:08:57Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (6659 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (6659 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240501 14:08:57.073021 6472 test_util.cc:268] Using random seed: 117673335
I20240501 14:08:57.073753 6472 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:57.073796 6472 ts_itest-base.cc:113] --------------
I20240501 14:08:57.073817 6472 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:08:57.073832 6472 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:08:57.073848 6472 ts_itest-base.cc:116] --------------
2024-05-01T14:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:57Z Disabled control of system clock
I20240501 14:08:57.081717 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:39223
--webserver_interface=127.6.82.62
--webserver_port=0
--builtin_ntp_servers=127.6.82.20:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.82.62:39223
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:08:57.186597 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.186851 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.186918 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.189846 7920 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.189920 7920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:57.189960 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.189997 7920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:57.190035 7920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:57.193596 7920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.82.62:39223
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:39223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.82.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.194473 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.195576 7920 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.202452 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.202616 7920 server_base.cc:1034] running on GCE node
W20240501 14:08:57.202486 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.202800 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.203307 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.204209 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.205399 7920 hybrid_clock.cc:648] HybridClock initialized: now 1714572537205376 us; error 31 us; skew 500 ppm
I20240501 14:08:57.207302 7920 webserver.cc:460] Webserver started at http://127.6.82.62:46803/ using document root <none> and password file <none>
I20240501 14:08:57.207851 7920 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.207913 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.208096 7920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:57.209718 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "66d81fe4747a4655b23ec12b260dc140"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.210177 7920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "66d81fe4747a4655b23ec12b260dc140"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.213529 7920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:57.215683 7934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.216712 7920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:57.216842 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
uuid: "66d81fe4747a4655b23ec12b260dc140"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.216923 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.228871 7920 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.229430 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.229596 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.235599 7920 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.62:39223
I20240501 14:08:57.235605 7986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.62:39223 every 8 connection(s)
I20240501 14:08:57.236552 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:57.239068 7987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:57.243777 7987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140: Bootstrap starting.
I20240501 14:08:57.245846 7987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:57.246255 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 7920
I20240501 14:08:57.246369 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:57.246634 7987 log.cc:824] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:57.248513 7987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140: No bootstrap required, opened a new log
I20240501 14:08:57.251211 7987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d81fe4747a4655b23ec12b260dc140" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 39223 } }
I20240501 14:08:57.251391 7987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:57.251467 7987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d81fe4747a4655b23ec12b260dc140, State: Initialized, Role: FOLLOWER
I20240501 14:08:57.251924 7987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d81fe4747a4655b23ec12b260dc140" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 39223 } }
I20240501 14:08:57.252097 7987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:57.252197 7987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:57.252295 7987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:57.352241 7987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d81fe4747a4655b23ec12b260dc140" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 39223 } }
I20240501 14:08:57.352710 7987 leader_election.cc:302] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d81fe4747a4655b23ec12b260dc140; no voters:
I20240501 14:08:57.353029 7987 leader_election.cc:288] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:57.353339 7992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:57.353677 7992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [term 1 LEADER]: Becoming Leader. State: Replica: 66d81fe4747a4655b23ec12b260dc140, State: Running, Role: LEADER
I20240501 14:08:57.354070 7992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d81fe4747a4655b23ec12b260dc140" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 39223 } }
I20240501 14:08:57.354261 7987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:57.355865 7994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d81fe4747a4655b23ec12b260dc140. Latest consensus state: current_term: 1 leader_uuid: "66d81fe4747a4655b23ec12b260dc140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d81fe4747a4655b23ec12b260dc140" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 39223 } } }
I20240501 14:08:57.355989 7994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:57.355938 7993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d81fe4747a4655b23ec12b260dc140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d81fe4747a4655b23ec12b260dc140" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 39223 } } }
I20240501 14:08:57.356074 7993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140 [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:57.357041 8003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:57.360119 8003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:57.364872 8003 catalog_manager.cc:1347] Generated new cluster ID: 25ff86fa9ef743399adc6370980b2bf8
I20240501 14:08:57.364928 8003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:57.373008 8003 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:57.373965 8003 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:57.380898 8003 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66d81fe4747a4655b23ec12b260dc140: Generated new TSK 0
I20240501 14:08:57.381481 8003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:57.392750 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.1:0
--local_ip_for_outbound_sockets=127.6.82.1
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--builtin_ntp_servers=127.6.82.20:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:08:57.502030 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.502265 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.502336 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.505317 8011 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.505403 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.505518 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.1
I20240501 14:08:57.509104 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--never_fsync=true
--heap_profile_path=/tmp/kudu.8011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.1
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.509974 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.511253 8011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.519306 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.519351 8011 server_base.cc:1034] running on GCE node
W20240501 14:08:57.519292 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.519542 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.519897 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.520411 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.521581 8011 hybrid_clock.cc:648] HybridClock initialized: now 1714572537521553 us; error 36 us; skew 500 ppm
I20240501 14:08:57.523538 8011 webserver.cc:460] Webserver started at http://127.6.82.1:36619/ using document root <none> and password file <none>
I20240501 14:08:57.524040 8011 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.524101 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.524251 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:57.525748 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5c5ebdbce09d4a248c2cb3d3799eee53"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.526172 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5c5ebdbce09d4a248c2cb3d3799eee53"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.529214 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:57.531248 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.532163 8011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.532249 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5c5ebdbce09d4a248c2cb3d3799eee53"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.532317 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.547154 8011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.547627 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.547773 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.548218 8011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.549046 8011 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.549098 8011 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.549145 8011 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.549167 8011 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.557260 8011 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.1:37955
I20240501 14:08:57.557281 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.1:37955 every 8 connection(s)
I20240501 14:08:57.558540 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:57.566834 8139 heartbeater.cc:347] Connected to a master server at 127.6.82.62:39223
I20240501 14:08:57.567046 8139 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.567551 8139 heartbeater.cc:511] Master 127.6.82.62:39223 requested a full tablet report, sending...
I20240501 14:08:57.568740 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8011
I20240501 14:08:57.568831 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:57.568818 7951 ts_manager.cc:194] Registered new tserver with Master: 5c5ebdbce09d4a248c2cb3d3799eee53 (127.6.82.1:37955)
I20240501 14:08:57.570026 7951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.1:41809
I20240501 14:08:57.571161 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.2:0
--local_ip_for_outbound_sockets=127.6.82.2
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--builtin_ntp_servers=127.6.82.20:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:08:57.677098 8142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.677331 8142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.677376 8142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.680341 8142 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.680403 8142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.680503 8142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.2
I20240501 14:08:57.683951 8142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--never_fsync=true
--heap_profile_path=/tmp/kudu.8142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.2
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.684779 8142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.685834 8142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.693310 8147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.693281 8150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.693298 8148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.693444 8142 server_base.cc:1034] running on GCE node
I20240501 14:08:57.693914 8142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.694422 8142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.695571 8142 hybrid_clock.cc:648] HybridClock initialized: now 1714572537695566 us; error 31 us; skew 500 ppm
I20240501 14:08:57.697364 8142 webserver.cc:460] Webserver started at http://127.6.82.2:35221/ using document root <none> and password file <none>
I20240501 14:08:57.697896 8142 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.697975 8142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.698159 8142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:57.699831 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "174addc281ec4863bfe0aad4bc6e305e"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.700296 8142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "174addc281ec4863bfe0aad4bc6e305e"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.703606 8142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:08:57.705729 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.706663 8142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.706802 8142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "174addc281ec4863bfe0aad4bc6e305e"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.706905 8142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.719817 8142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.720381 8142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.720530 8142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.721035 8142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.721887 8142 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.721938 8142 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.722009 8142 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.722049 8142 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.730126 8142 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.2:46041
I20240501 14:08:57.730144 8269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.2:46041 every 8 connection(s)
I20240501 14:08:57.731135 8142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:08:57.736788 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8142
I20240501 14:08:57.736922 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:08:57.738981 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.3:0
--local_ip_for_outbound_sockets=127.6.82.3
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--builtin_ntp_servers=127.6.82.20:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:57.739681 8270 heartbeater.cc:347] Connected to a master server at 127.6.82.62:39223
I20240501 14:08:57.739907 8270 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.740506 8270 heartbeater.cc:511] Master 127.6.82.62:39223 requested a full tablet report, sending...
I20240501 14:08:57.741452 7951 ts_manager.cc:194] Registered new tserver with Master: 174addc281ec4863bfe0aad4bc6e305e (127.6.82.2:46041)
I20240501 14:08:57.741991 7951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.2:52839
W20240501 14:08:57.845064 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:57.845319 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:57.845387 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:57.848333 8273 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:57.848404 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:57.848495 8273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.3
I20240501 14:08:57.852454 8273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.3
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:57.853324 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:57.854425 8273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:57.862047 8279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.862066 8281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:57.862111 8278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:57.862399 8273 server_base.cc:1034] running on GCE node
I20240501 14:08:57.862704 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:57.863238 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:57.864379 8273 hybrid_clock.cc:648] HybridClock initialized: now 1714572537864366 us; error 32 us; skew 500 ppm
I20240501 14:08:57.866068 8273 webserver.cc:460] Webserver started at http://127.6.82.3:40299/ using document root <none> and password file <none>
I20240501 14:08:57.866663 8273 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:57.866775 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:57.866959 8273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:57.868685 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7b49a649aa46402fadb9484d6badeeb7"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.869172 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7b49a649aa46402fadb9484d6badeeb7"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.872375 8273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:08:57.874534 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.875478 8273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:57.875607 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b49a649aa46402fadb9484d6badeeb7"
format_stamp: "Formatted at 2024-05-01 14:08:57 on dist-test-slave-crgf"
I20240501 14:08:57.875687 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:57.901134 8273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:57.901783 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:57.901948 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:57.902505 8273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:57.903434 8273 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:57.903484 8273 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.903544 8273 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:57.903576 8273 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:57.911830 8273 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.3:46023
I20240501 14:08:57.911877 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.3:46023 every 8 connection(s)
I20240501 14:08:57.912724 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:08:57.915294 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8273
I20240501 14:08:57.915400 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:08:57.918421 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.4:0
--local_ip_for_outbound_sockets=127.6.82.4
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--builtin_ntp_servers=127.6.82.20:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:57.922317 8401 heartbeater.cc:347] Connected to a master server at 127.6.82.62:39223
I20240501 14:08:57.922534 8401 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:57.923162 8401 heartbeater.cc:511] Master 127.6.82.62:39223 requested a full tablet report, sending...
I20240501 14:08:57.924219 7951 ts_manager.cc:194] Registered new tserver with Master: 7b49a649aa46402fadb9484d6badeeb7 (127.6.82.3:46023)
I20240501 14:08:57.925053 7951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.3:42053
W20240501 14:08:58.024732 8404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:58.024950 8404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:58.024991 8404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:58.027910 8404 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:58.027968 8404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:58.028031 8404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.4
I20240501 14:08:58.031548 8404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--never_fsync=true
--heap_profile_path=/tmp/kudu.8404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.4
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:58.032390 8404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:58.033440 8404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:58.041186 8409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:58.041193 8410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:58.041438 8404 server_base.cc:1034] running on GCE node
W20240501 14:08:58.041481 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:58.041849 8404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:58.042336 8404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:58.043485 8404 hybrid_clock.cc:648] HybridClock initialized: now 1714572538043466 us; error 37 us; skew 500 ppm
I20240501 14:08:58.045259 8404 webserver.cc:460] Webserver started at http://127.6.82.4:40867/ using document root <none> and password file <none>
I20240501 14:08:58.045782 8404 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:58.045836 8404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:58.046015 8404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:58.047621 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2b29b79d52aa44a3b15083c60a15bed0"
format_stamp: "Formatted at 2024-05-01 14:08:58 on dist-test-slave-crgf"
I20240501 14:08:58.048069 8404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2b29b79d52aa44a3b15083c60a15bed0"
format_stamp: "Formatted at 2024-05-01 14:08:58 on dist-test-slave-crgf"
I20240501 14:08:58.051187 8404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:58.053189 8418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:58.054152 8404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:58.054244 8404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2b29b79d52aa44a3b15083c60a15bed0"
format_stamp: "Formatted at 2024-05-01 14:08:58 on dist-test-slave-crgf"
I20240501 14:08:58.054327 8404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:58.064627 8404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:58.065135 8404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:58.065279 8404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:58.065763 8404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:58.066669 8404 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:58.066715 8404 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:58.066780 8404 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:58.066815 8404 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:58.074582 8404 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.4:41481
I20240501 14:08:58.074606 8531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.4:41481 every 8 connection(s)
I20240501 14:08:58.075529 8404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:08:58.083704 8532 heartbeater.cc:347] Connected to a master server at 127.6.82.62:39223
I20240501 14:08:58.083920 8532 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:58.084203 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8404
I20240501 14:08:58.084312 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:08:58.084544 8532 heartbeater.cc:511] Master 127.6.82.62:39223 requested a full tablet report, sending...
I20240501 14:08:58.085492 7950 ts_manager.cc:194] Registered new tserver with Master: 2b29b79d52aa44a3b15083c60a15bed0 (127.6.82.4:41481)
I20240501 14:08:58.086088 7950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.4:37595
I20240501 14:08:58.086350 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.5:0
--local_ip_for_outbound_sockets=127.6.82.5
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--builtin_ntp_servers=127.6.82.20:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:08:58.189011 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:58.189229 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:58.189275 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:58.192226 8535 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:58.192286 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:58.192385 8535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.5
I20240501 14:08:58.195816 8535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:39223
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.5
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:58.196641 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:58.197702 8535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:58.205273 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:58.205288 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:58.205265 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:58.205550 8535 server_base.cc:1034] running on GCE node
I20240501 14:08:58.205937 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:58.206463 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:58.207608 8535 hybrid_clock.cc:648] HybridClock initialized: now 1714572538207586 us; error 51 us; skew 500 ppm
I20240501 14:08:58.209357 8535 webserver.cc:460] Webserver started at http://127.6.82.5:34607/ using document root <none> and password file <none>
I20240501 14:08:58.209893 8535 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:58.209954 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:58.210120 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:58.211731 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "17701f1d836f41e68677130f1d776840"
format_stamp: "Formatted at 2024-05-01 14:08:58 on dist-test-slave-crgf"
I20240501 14:08:58.212182 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "17701f1d836f41e68677130f1d776840"
format_stamp: "Formatted at 2024-05-01 14:08:58 on dist-test-slave-crgf"
I20240501 14:08:58.215250 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:08:58.217303 8549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:58.218223 8535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:08:58.218338 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "17701f1d836f41e68677130f1d776840"
format_stamp: "Formatted at 2024-05-01 14:08:58 on dist-test-slave-crgf"
I20240501 14:08:58.218417 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:58.228950 8535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:58.229445 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:58.229606 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:58.230096 8535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:58.230940 8535 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:58.230990 8535 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:58.231046 8535 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:58.231073 8535 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:58.239486 8535 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.5:32933
I20240501 14:08:58.239694 8662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.5:32933 every 8 connection(s)
I20240501 14:08:58.240449 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:08:58.241305 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8535
I20240501 14:08:58.241407 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:08:58.249500 8663 heartbeater.cc:347] Connected to a master server at 127.6.82.62:39223
I20240501 14:08:58.249697 8663 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:58.250145 8663 heartbeater.cc:511] Master 127.6.82.62:39223 requested a full tablet report, sending...
I20240501 14:08:58.251048 7950 ts_manager.cc:194] Registered new tserver with Master: 17701f1d836f41e68677130f1d776840 (127.6.82.5:32933)
I20240501 14:08:58.251566 7950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.5:43993
I20240501 14:08:58.255398 6472 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:58.263657 7950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40750:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:08:58.281220 8073 tablet_service.cc:1467] Processing CreateTablet for tablet dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 (DEFAULT_TABLE table=TestTable [id=3708168c0f35488782261dacedcfb474]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.281270 8597 tablet_service.cc:1467] Processing CreateTablet for tablet dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 (DEFAULT_TABLE table=TestTable [id=3708168c0f35488782261dacedcfb474]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.281625 8335 tablet_service.cc:1467] Processing CreateTablet for tablet dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 (DEFAULT_TABLE table=TestTable [id=3708168c0f35488782261dacedcfb474]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:08:58.282189 8597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.282189 8073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.282325 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:58.287084 8680 tablet_bootstrap.cc:492] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: Bootstrap starting.
I20240501 14:08:58.288107 8682 tablet_bootstrap.cc:492] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: Bootstrap starting.
I20240501 14:08:58.288182 8681 tablet_bootstrap.cc:492] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: Bootstrap starting.
I20240501 14:08:58.289712 8680 tablet_bootstrap.cc:654] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.289753 8681 tablet_bootstrap.cc:654] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.289862 8682 tablet_bootstrap.cc:654] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:58.290488 8681 log.cc:824] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:58.290591 8682 log.cc:824] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:58.290649 8680 log.cc:824] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:58.292078 8680 tablet_bootstrap.cc:492] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: No bootstrap required, opened a new log
I20240501 14:08:58.292147 8681 tablet_bootstrap.cc:492] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: No bootstrap required, opened a new log
I20240501 14:08:58.292161 8682 tablet_bootstrap.cc:492] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: No bootstrap required, opened a new log
I20240501 14:08:58.292239 8680 ts_tablet_manager.cc:1367] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:08:58.292302 8682 ts_tablet_manager.cc:1367] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:08:58.292304 8681 ts_tablet_manager.cc:1367] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:08:58.294903 8680 raft_consensus.cc:348] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.295071 8680 raft_consensus.cc:374] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.295305 8680 raft_consensus.cc:729] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b49a649aa46402fadb9484d6badeeb7, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.295433 8681 raft_consensus.cc:348] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.295671 8681 raft_consensus.cc:374] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.295740 8681 raft_consensus.cc:729] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5ebdbce09d4a248c2cb3d3799eee53, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.295804 8680 consensus_queue.cc:260] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.295770 8682 raft_consensus.cc:348] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.295934 8682 raft_consensus.cc:374] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:58.296010 8682 raft_consensus.cc:729] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17701f1d836f41e68677130f1d776840, State: Initialized, Role: FOLLOWER
I20240501 14:08:58.296175 8681 consensus_queue.cc:260] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.296416 8682 consensus_queue.cc:260] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.296656 8680 ts_tablet_manager.cc:1398] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 14:08:58.296856 8139 heartbeater.cc:503] Master 127.6.82.62:39223 was elected leader, sending a full tablet report...
I20240501 14:08:58.297032 8681 ts_tablet_manager.cc:1398] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:08:58.297150 8682 ts_tablet_manager.cc:1398] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 14:08:58.297480 8663 heartbeater.cc:503] Master 127.6.82.62:39223 was elected leader, sending a full tablet report...
I20240501 14:08:58.297802 8401 heartbeater.cc:503] Master 127.6.82.62:39223 was elected leader, sending a full tablet report...
W20240501 14:08:58.310056 8140 tablet.cc:2369] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:58.413808 8402 tablet.cc:2369] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:08:58.491312 8664 tablet.cc:2369] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:08:58.631148 8687 raft_consensus.cc:482] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:58.631419 8687 raft_consensus.cc:504] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.632755 8687 leader_election.cc:288] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17701f1d836f41e68677130f1d776840 (127.6.82.5:32933), 7b49a649aa46402fadb9484d6badeeb7 (127.6.82.3:46023)
I20240501 14:08:58.637825 8617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3" candidate_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17701f1d836f41e68677130f1d776840" is_pre_election: true
I20240501 14:08:58.637822 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3" candidate_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49a649aa46402fadb9484d6badeeb7" is_pre_election: true
I20240501 14:08:58.638109 8617 raft_consensus.cc:2454] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c5ebdbce09d4a248c2cb3d3799eee53 in term 0.
I20240501 14:08:58.638109 8355 raft_consensus.cc:2454] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c5ebdbce09d4a248c2cb3d3799eee53 in term 0.
I20240501 14:08:58.638582 8027 leader_election.cc:302] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17701f1d836f41e68677130f1d776840, 5c5ebdbce09d4a248c2cb3d3799eee53, 7b49a649aa46402fadb9484d6badeeb7; no voters:
I20240501 14:08:58.638872 8687 raft_consensus.cc:2781] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:08:58.638952 8687 raft_consensus.cc:482] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:08:58.638993 8687 raft_consensus.cc:3037] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.639925 8687 raft_consensus.cc:504] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.640282 8687 leader_election.cc:288] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [CANDIDATE]: Term 1 election: Requested vote from peers 17701f1d836f41e68677130f1d776840 (127.6.82.5:32933), 7b49a649aa46402fadb9484d6badeeb7 (127.6.82.3:46023)
I20240501 14:08:58.640632 8617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3" candidate_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17701f1d836f41e68677130f1d776840"
I20240501 14:08:58.640650 8355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3" candidate_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b49a649aa46402fadb9484d6badeeb7"
I20240501 14:08:58.640749 8617 raft_consensus.cc:3037] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.640780 8355 raft_consensus.cc:3037] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:58.641608 8355 raft_consensus.cc:2454] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 7b49a649aa46402fadb9484d6badeeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c5ebdbce09d4a248c2cb3d3799eee53 in term 1.
I20240501 14:08:58.641608 8617 raft_consensus.cc:2454] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 17701f1d836f41e68677130f1d776840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c5ebdbce09d4a248c2cb3d3799eee53 in term 1.
I20240501 14:08:58.641913 8027 leader_election.cc:302] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17701f1d836f41e68677130f1d776840, 5c5ebdbce09d4a248c2cb3d3799eee53, 7b49a649aa46402fadb9484d6badeeb7; no voters:
I20240501 14:08:58.642062 8687 raft_consensus.cc:2781] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:58.642257 8687 raft_consensus.cc:686] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [term 1 LEADER]: Becoming Leader. State: Replica: 5c5ebdbce09d4a248c2cb3d3799eee53, State: Running, Role: LEADER
I20240501 14:08:58.642561 8687 consensus_queue.cc:237] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } }
I20240501 14:08:58.645519 7951 catalog_manager.cc:5484] T dbdd5d9ca0ad4b3b8f1c14fd55e6a6e3 P 5c5ebdbce09d4a248c2cb3d3799eee53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c5ebdbce09d4a248c2cb3d3799eee53 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c5ebdbce09d4a248c2cb3d3799eee53" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 37955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17701f1d836f41e68677130f1d776840" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 32933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b49a649aa46402fadb9484d6badeeb7" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 46023 } health_report { overall_health: UNKNOWN } } }
I20240501 14:08:58.724337 6472 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:08:58.725775 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5c5ebdbce09d4a248c2cb3d3799eee53 to finish bootstrapping
I20240501 14:08:58.730759 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 174addc281ec4863bfe0aad4bc6e305e to finish bootstrapping
I20240501 14:08:58.736611 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7b49a649aa46402fadb9484d6badeeb7 to finish bootstrapping
I20240501 14:08:58.740778 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2b29b79d52aa44a3b15083c60a15bed0 to finish bootstrapping
I20240501 14:08:58.743880 8270 heartbeater.cc:503] Master 127.6.82.62:39223 was elected leader, sending a full tablet report...
I20240501 14:08:58.745960 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 17701f1d836f41e68677130f1d776840 to finish bootstrapping
I20240501 14:08:58.750231 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8011
I20240501 14:08:58.932336 8203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.932406 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.932505 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:58.933770 8597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.009322 8203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.009438 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.009734 8597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.015883 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.088311 8532 heartbeater.cc:503] Master 127.6.82.62:39223 was elected leader, sending a full tablet report...
I20240501 14:08:59.115474 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.115692 8203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.115969 8597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.116494 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.326475 8597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.326501 8203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.327838 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.328357 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.397166 8597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.398094 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.398157 8203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.398342 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.496080 8597 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.496101 8203 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.496913 8466 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.497714 8335 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:08:59.529387 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8142
I20240501 14:08:59.534906 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8273
I20240501 14:08:59.540503 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8404
I20240501 14:08:59.545888 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8535
I20240501 14:08:59.551131 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 7920
2024-05-01T14:08:59Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2500 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2500 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240501 14:08:59.573629 6472 test_util.cc:268] Using random seed: 120173934
I20240501 14:08:59.574404 6472 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:08:59.574447 6472 ts_itest-base.cc:113] --------------
I20240501 14:08:59.574458 6472 ts_itest-base.cc:114] 7 tablet servers
I20240501 14:08:59.574468 6472 ts_itest-base.cc:115] 1 replicas per TS
I20240501 14:08:59.574478 6472 ts_itest-base.cc:116] --------------
2024-05-01T14:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:08:59Z Disabled control of system clock
I20240501 14:08:59.582177 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:37407
--webserver_interface=127.6.82.62
--webserver_port=0
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.82.62:37407
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 14:08:59.683641 8879 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:08:59.683876 8879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:59.683925 8879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:59.683995 8879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:59.686902 8879 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:59.686995 8879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:08:59.687036 8879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:59.687072 8879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:08:59.687111 8879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:08:59.690639 8879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.82.62:37407
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.82.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:59.691471 8879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:59.692514 8879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:59.699193 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.699208 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.699208 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.699512 8879 server_base.cc:1034] running on GCE node
I20240501 14:08:59.699965 8879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:59.700783 8879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:59.701970 8879 hybrid_clock.cc:648] HybridClock initialized: now 1714572539701943 us; error 39 us; skew 500 ppm
I20240501 14:08:59.703791 8879 webserver.cc:460] Webserver started at http://127.6.82.62:40519/ using document root <none> and password file <none>
I20240501 14:08:59.704325 8879 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:59.704380 8879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:59.704555 8879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:59.706121 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-crgf"
I20240501 14:08:59.706597 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-crgf"
I20240501 14:08:59.709731 8879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:08:59.711835 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.712785 8879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:08:59.712888 8879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-crgf"
I20240501 14:08:59.712962 8879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:59.727167 8879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:59.727785 8879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:59.727934 8879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:59.734180 8879 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.62:37407
I20240501 14:08:59.734185 8945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.62:37407 every 8 connection(s)
I20240501 14:08:59.735103 8879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:08:59.736917 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8879
I20240501 14:08:59.737030 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:08:59.737730 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:08:59.743896 8946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb: Bootstrap starting.
I20240501 14:08:59.746145 8946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb: Neither blocks nor log segments found. Creating new log.
I20240501 14:08:59.746927 8946 log.cc:824] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:08:59.748310 8946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb: No bootstrap required, opened a new log
I20240501 14:08:59.750572 8946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 37407 } }
I20240501 14:08:59.750766 8946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:08:59.750818 8946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6308ae94f7da4aa6bb70f8e9cb63fceb, State: Initialized, Role: FOLLOWER
I20240501 14:08:59.751170 8946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 37407 } }
I20240501 14:08:59.751269 8946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:08:59.751312 8946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:08:59.751384 8946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:08:59.752039 8946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 37407 } }
I20240501 14:08:59.752336 8946 leader_election.cc:302] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6308ae94f7da4aa6bb70f8e9cb63fceb; no voters:
I20240501 14:08:59.752633 8946 leader_election.cc:288] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:08:59.753062 8951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:08:59.753253 8951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [term 1 LEADER]: Becoming Leader. State: Replica: 6308ae94f7da4aa6bb70f8e9cb63fceb, State: Running, Role: LEADER
I20240501 14:08:59.753513 8951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 37407 } }
I20240501 14:08:59.753661 8946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:08:59.755025 8953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6308ae94f7da4aa6bb70f8e9cb63fceb. Latest consensus state: current_term: 1 leader_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 37407 } } }
I20240501 14:08:59.755061 8952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6308ae94f7da4aa6bb70f8e9cb63fceb" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 37407 } } }
I20240501 14:08:59.755139 8953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:59.755162 8952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb [sys.catalog]: This master's current role is: LEADER
I20240501 14:08:59.755446 8961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:08:59.757797 8961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:08:59.762238 8961 catalog_manager.cc:1347] Generated new cluster ID: fbc90113654a4051a3132e179a168716
I20240501 14:08:59.762305 8961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:08:59.774386 8961 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:08:59.775373 8961 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:08:59.784317 8961 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6308ae94f7da4aa6bb70f8e9cb63fceb: Generated new TSK 0
I20240501 14:08:59.784956 8961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:08:59.794016 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.1:0
--local_ip_for_outbound_sockets=127.6.82.1
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:08:59.898594 8970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:08:59.898839 8970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:08:59.898890 8970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:08:59.901849 8970 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:08:59.901908 8970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:08:59.901985 8970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.1
I20240501 14:08:59.905519 8970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.8970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.1
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:08:59.906325 8970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:08:59.907460 8970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:08:59.914820 8976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:08:59.914788 8978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.914925 8970 server_base.cc:1034] running on GCE node
W20240501 14:08:59.914799 8975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:08:59.915376 8970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:08:59.915886 8970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:08:59.917043 8970 hybrid_clock.cc:648] HybridClock initialized: now 1714572539917026 us; error 31 us; skew 500 ppm
I20240501 14:08:59.918870 8970 webserver.cc:460] Webserver started at http://127.6.82.1:46227/ using document root <none> and password file <none>
I20240501 14:08:59.919378 8970 fs_manager.cc:359] Metadata directory not provided
I20240501 14:08:59.919447 8970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:08:59.919641 8970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:08:59.921221 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b4c1a584bc6542f68b6d6912b091f1af"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-crgf"
I20240501 14:08:59.921690 8970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b4c1a584bc6542f68b6d6912b091f1af"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-crgf"
I20240501 14:08:59.924883 8970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 14:08:59.926990 8984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.927948 8970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:08:59.928061 8970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b4c1a584bc6542f68b6d6912b091f1af"
format_stamp: "Formatted at 2024-05-01 14:08:59 on dist-test-slave-crgf"
I20240501 14:08:59.928148 8970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:08:59.941965 8970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:08:59.942543 8970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:08:59.942698 8970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:08:59.943223 8970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:08:59.944059 8970 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:08:59.944103 8970 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.944162 8970 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:08:59.944190 8970 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:08:59.952237 8970 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.1:43063
I20240501 14:08:59.952265 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.1:43063 every 8 connection(s)
I20240501 14:08:59.953085 8970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:08:59.959599 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 8970
I20240501 14:08:59.959687 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:08:59.961268 9098 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:08:59.961494 9098 heartbeater.cc:464] Registering TS with master...
I20240501 14:08:59.961760 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.2:0
--local_ip_for_outbound_sockets=127.6.82.2
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:08:59.962106 9098 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:08:59.963534 8910 ts_manager.cc:194] Registered new tserver with Master: b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:08:59.964658 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.1:44135
W20240501 14:09:00.066576 9101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.066819 9101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:00.066882 9101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.069945 9101 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:00.070017 9101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:00.070223 9101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.2
I20240501 14:09:00.073792 9101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.9101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.2
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:00.074707 9101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:00.075930 9101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:00.083392 9109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.083547 9107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.083561 9106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.083904 9101 server_base.cc:1034] running on GCE node
I20240501 14:09:00.084219 9101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:00.084761 9101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:00.085908 9101 hybrid_clock.cc:648] HybridClock initialized: now 1714572540085899 us; error 32 us; skew 500 ppm
I20240501 14:09:00.087726 9101 webserver.cc:460] Webserver started at http://127.6.82.2:36977/ using document root <none> and password file <none>
I20240501 14:09:00.088232 9101 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.088291 9101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.088457 9101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.090005 9101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "90e73913f35c45dd8f2f517c3a385d3a"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.090445 9101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "90e73913f35c45dd8f2f517c3a385d3a"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.093693 9101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:00.095749 9115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.096657 9101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:00.096769 9101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "90e73913f35c45dd8f2f517c3a385d3a"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.096850 9101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.114811 9101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.115444 9101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:00.115602 9101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.116124 9101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.117027 9101 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.117079 9101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.117139 9101 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.117165 9101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.125344 9101 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.2:37403
I20240501 14:09:00.125365 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.2:37403 every 8 connection(s)
I20240501 14:09:00.126231 9101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:00.127424 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 9101
I20240501 14:09:00.127523 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:00.129635 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.3:0
--local_ip_for_outbound_sockets=127.6.82.3
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:00.135807 9229 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:00.136058 9229 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.136734 9229 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:00.137794 8910 ts_manager.cc:194] Registered new tserver with Master: 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:00.138351 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.2:45355
W20240501 14:09:00.234421 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.234629 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:00.234675 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.237756 9232 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:00.237815 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:00.237906 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.3
I20240501 14:09:00.241446 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.9232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.3
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:00.242286 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:00.243395 9232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:00.250850 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.250846 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.250939 9232 server_base.cc:1034] running on GCE node
W20240501 14:09:00.250846 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.251390 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:00.251902 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:00.253067 9232 hybrid_clock.cc:648] HybridClock initialized: now 1714572540253056 us; error 26 us; skew 500 ppm
I20240501 14:09:00.255136 9232 webserver.cc:460] Webserver started at http://127.6.82.3:34025/ using document root <none> and password file <none>
I20240501 14:09:00.255759 9232 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.255851 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.256055 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.257719 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f2040b402c764007a4b1d969db917ede"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.258213 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f2040b402c764007a4b1d969db917ede"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.261610 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240501 14:09:00.263841 9246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.264840 9232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:00.264943 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f2040b402c764007a4b1d969db917ede"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.265022 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.279657 9232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.280277 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:00.280438 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.280966 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.281816 9232 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.281873 9232 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.281935 9232 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.281962 9232 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.290518 9232 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.3:37685
I20240501 14:09:00.290566 9359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.3:37685 every 8 connection(s)
I20240501 14:09:00.291455 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:00.295254 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 9232
I20240501 14:09:00.295353 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:00.297405 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.4:0
--local_ip_for_outbound_sockets=127.6.82.4
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:00.300204 9360 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:00.300410 9360 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.300895 9360 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:00.301954 8910 ts_manager.cc:194] Registered new tserver with Master: f2040b402c764007a4b1d969db917ede (127.6.82.3:37685)
I20240501 14:09:00.302809 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.3:59799
W20240501 14:09:00.405030 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.405215 9363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:00.405254 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.408159 9363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:00.408214 9363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:00.408274 9363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.4
I20240501 14:09:00.412554 9363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.9363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.4
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:00.413347 9363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:00.414368 9363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:00.421767 9369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.421785 9368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.421789 9371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.422044 9363 server_base.cc:1034] running on GCE node
I20240501 14:09:00.422389 9363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:00.422946 9363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:00.424136 9363 hybrid_clock.cc:648] HybridClock initialized: now 1714572540424119 us; error 33 us; skew 500 ppm
I20240501 14:09:00.425966 9363 webserver.cc:460] Webserver started at http://127.6.82.4:44769/ using document root <none> and password file <none>
I20240501 14:09:00.426522 9363 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.426589 9363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.426813 9363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.428457 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fa75a3f967ef4d259a6e56f2b457b39b"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.428947 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fa75a3f967ef4d259a6e56f2b457b39b"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.432433 9363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:00.434505 9377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.435482 9363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:00.435587 9363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fa75a3f967ef4d259a6e56f2b457b39b"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.435688 9363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.467975 9363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.468634 9363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:00.468811 9363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.469384 9363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.470276 9363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.470324 9363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.470369 9363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.470402 9363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.479277 9363 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.4:33949
I20240501 14:09:00.479307 9490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.4:33949 every 8 connection(s)
I20240501 14:09:00.480180 9363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:00.483659 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 9363
I20240501 14:09:00.483783 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:00.485893 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.5:0
--local_ip_for_outbound_sockets=127.6.82.5
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:00.488971 9491 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:00.489168 9491 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.489645 9491 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:00.490618 8910 ts_manager.cc:194] Registered new tserver with Master: fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4:33949)
I20240501 14:09:00.491432 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.4:56943
W20240501 14:09:00.589401 9494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.589601 9494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:00.589641 9494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.592597 9494 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:00.592655 9494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:00.592716 9494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.5
I20240501 14:09:00.596113 9494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.9494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.5
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:00.596889 9494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:00.597923 9494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:00.605737 9502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.605731 9499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.605950 9500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.606235 9494 server_base.cc:1034] running on GCE node
I20240501 14:09:00.606539 9494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:00.607059 9494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:00.608198 9494 hybrid_clock.cc:648] HybridClock initialized: now 1714572540608178 us; error 40 us; skew 500 ppm
I20240501 14:09:00.609951 9494 webserver.cc:460] Webserver started at http://127.6.82.5:36645/ using document root <none> and password file <none>
I20240501 14:09:00.610456 9494 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.610512 9494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.610697 9494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.612320 9494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b42390c48f8c4e71a476d98115a685ac"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.612740 9494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b42390c48f8c4e71a476d98115a685ac"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.615777 9494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:09:00.617883 9508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.618841 9494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:00.618935 9494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b42390c48f8c4e71a476d98115a685ac"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.619016 9494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.631726 9494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.632339 9494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:00.632503 9494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.633039 9494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.633888 9494 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.633937 9494 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.633994 9494 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.634025 9494 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.642642 9494 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.5:46807
I20240501 14:09:00.642688 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.5:46807 every 8 connection(s)
I20240501 14:09:00.643566 9494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:00.651638 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 9494
I20240501 14:09:00.651731 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:00.653833 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.6:0
--local_ip_for_outbound_sockets=127.6.82.6
--webserver_interface=127.6.82.6
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:00.660284 9622 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:00.660519 9622 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.661087 9622 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:00.662094 8910 ts_manager.cc:194] Registered new tserver with Master: b42390c48f8c4e71a476d98115a685ac (127.6.82.5:46807)
I20240501 14:09:00.662895 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.5:52259
W20240501 14:09:00.763505 9625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.763741 9625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:00.763787 9625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.766712 9625 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:00.766809 9625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:00.766906 9625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.6
I20240501 14:09:00.770480 9625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.82.6
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.9625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.6
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:00.771370 9625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:00.772471 9625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:00.779847 9630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.779891 9631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.779985 9625 server_base.cc:1034] running on GCE node
W20240501 14:09:00.779995 9633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.780421 9625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:00.780901 9625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:00.782037 9625 hybrid_clock.cc:648] HybridClock initialized: now 1714572540782008 us; error 42 us; skew 500 ppm
I20240501 14:09:00.783780 9625 webserver.cc:460] Webserver started at http://127.6.82.6:36697/ using document root <none> and password file <none>
I20240501 14:09:00.784271 9625 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.784322 9625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.784493 9625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.786008 9625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e32ae6129e5a47aa955c480ff1962e06"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.786468 9625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e32ae6129e5a47aa955c480ff1962e06"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.789553 9625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 14:09:00.791688 9639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.792610 9625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:00.792732 9625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e32ae6129e5a47aa955c480ff1962e06"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.792812 9625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:00.829262 9625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:00.829941 9625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:00.830106 9625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:00.830655 9625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:00.831610 9625 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:00.831669 9625 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.831734 9625 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:00.831761 9625 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.840257 9625 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.6:34217
I20240501 14:09:00.840297 9752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.6:34217 every 8 connection(s)
I20240501 14:09:00.841200 9625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:00.849526 9753 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:00.849642 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 9625
I20240501 14:09:00.849749 9753 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:00.849758 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:09:00.850258 9753 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:00.851188 8910 ts_manager.cc:194] Registered new tserver with Master: e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6:34217)
I20240501 14:09:00.851768 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.6:43951
I20240501 14:09:00.851821 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.7:0
--local_ip_for_outbound_sockets=127.6.82.7
--webserver_interface=127.6.82.7
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:00.956110 9756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:00.956307 9756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:00.956349 9756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:00.959452 9756 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:00.959527 9756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:00.959621 9756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.7
I20240501 14:09:00.963446 9756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.82.7
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.9756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.7
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:00.964286 9756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:00.965377 9756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:09:00.966578 9098 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
W20240501 14:09:00.972963 9764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:00.973065 9761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.973158 9756 server_base.cc:1034] running on GCE node
W20240501 14:09:00.972970 9762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:00.973800 9756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:00.974383 9756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:00.975539 9756 hybrid_clock.cc:648] HybridClock initialized: now 1714572540975497 us; error 61 us; skew 500 ppm
I20240501 14:09:00.977366 9756 webserver.cc:460] Webserver started at http://127.6.82.7:40989/ using document root <none> and password file <none>
I20240501 14:09:00.977926 9756 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:00.977990 9756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:00.978307 9756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:00.979983 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "06295523e9de4e2dae4b0909b0a04ea6"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.980430 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "06295523e9de4e2dae4b0909b0a04ea6"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.983538 9756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:00.985591 9770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:00.986537 9756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:00.986642 9756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "06295523e9de4e2dae4b0909b0a04ea6"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:00.986721 9756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:01.011435 9756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:01.012094 9756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:01.012256 9756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:01.012799 9756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:01.013672 9756 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:01.013720 9756 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:01.013780 9756 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:01.013811 9756 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:01.022773 9756 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.7:42267
I20240501 14:09:01.022815 9883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.7:42267 every 8 connection(s)
I20240501 14:09:01.023676 9756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:01.027109 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 9756
I20240501 14:09:01.027218 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:09:01.032224 9884 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:01.032518 9884 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:01.033074 9884 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:01.033964 8910 ts_manager.cc:194] Registered new tserver with Master: 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7:42267)
I20240501 14:09:01.034499 8910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.7:47595
I20240501 14:09:01.041321 6472 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:09:01.049780 8910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59986:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:09:01.065999 9294 tablet_service.cc:1467] Processing CreateTablet for tablet 6a41ee08ad7141f5bb360dc992a3c072 (DEFAULT_TABLE table=TestTable [id=5f575adf81974c4c9ce0d657e3d3b833]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:01.066880 9294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a41ee08ad7141f5bb360dc992a3c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:01.071920 9898 tablet_bootstrap.cc:492] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Bootstrap starting.
I20240501 14:09:01.073730 9898 tablet_bootstrap.cc:654] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:01.074415 9898 log.cc:824] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:01.075984 9898 tablet_bootstrap.cc:492] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: No bootstrap required, opened a new log
I20240501 14:09:01.076156 9898 ts_tablet_manager.cc:1367] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:09:01.078574 9898 raft_consensus.cc:348] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:01.078783 9898 raft_consensus.cc:374] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:01.078851 9898 raft_consensus.cc:729] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2040b402c764007a4b1d969db917ede, State: Initialized, Role: FOLLOWER
I20240501 14:09:01.079272 9898 consensus_queue.cc:260] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:01.079394 9898 raft_consensus.cc:388] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:01.079463 9898 raft_consensus.cc:482] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:01.079571 9898 raft_consensus.cc:3037] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:01.080655 9898 raft_consensus.cc:504] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:01.080955 9898 leader_election.cc:302] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2040b402c764007a4b1d969db917ede; no voters:
I20240501 14:09:01.081245 9898 leader_election.cc:288] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:01.081496 9900 raft_consensus.cc:2781] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:01.081727 9900 raft_consensus.cc:686] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 LEADER]: Becoming Leader. State: Replica: f2040b402c764007a4b1d969db917ede, State: Running, Role: LEADER
I20240501 14:09:01.081897 9360 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:01.082072 9898 ts_tablet_manager.cc:1398] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240501 14:09:01.082034 9900 consensus_queue.cc:237] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:01.085606 8910 catalog_manager.cc:5484] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede reported cstate change: term changed from 0 to 1, leader changed from <none> to f2040b402c764007a4b1d969db917ede (127.6.82.3). New cstate: current_term: 1 leader_uuid: "f2040b402c764007a4b1d969db917ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:01.092326 6472 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:09:01.093783 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b4c1a584bc6542f68b6d6912b091f1af to finish bootstrapping
I20240501 14:09:01.099722 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 90e73913f35c45dd8f2f517c3a385d3a to finish bootstrapping
I20240501 14:09:01.105151 6472 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f2040b402c764007a4b1d969db917ede to finish bootstrapping
I20240501 14:09:01.109571 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fa75a3f967ef4d259a6e56f2b457b39b to finish bootstrapping
I20240501 14:09:01.115190 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b42390c48f8c4e71a476d98115a685ac to finish bootstrapping
I20240501 14:09:01.120951 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e32ae6129e5a47aa955c480ff1962e06 to finish bootstrapping
I20240501 14:09:01.132138 6472 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 06295523e9de4e2dae4b0909b0a04ea6 to finish bootstrapping
I20240501 14:09:01.137787 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 9232
I20240501 14:09:01.140584 9229 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:01.143690 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 9363
I20240501 14:09:01.149001 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 9494
I20240501 14:09:01.154250 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 9625
I20240501 14:09:01.159821 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 9756
I20240501 14:09:04.916424 8909 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59986:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:04.929118 9163 tablet_service.cc:1467] Processing CreateTablet for tablet dfb5160c34df401fb54fb75eab797b92 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ac384487a42c4c8aa5b9e8a68333b96f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:04.929118 9032 tablet_service.cc:1467] Processing CreateTablet for tablet e138c8048df144d39157f2918473e43d (DEFAULT_TABLE table=rebalance_test_table_0 [id=ac384487a42c4c8aa5b9e8a68333b96f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:04.929132 9031 tablet_service.cc:1467] Processing CreateTablet for tablet 7316a1b44e054c96acc41b645b2d33c4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ac384487a42c4c8aa5b9e8a68333b96f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:04.929955 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb5160c34df401fb54fb75eab797b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:04.930022 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7316a1b44e054c96acc41b645b2d33c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:04.930361 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e138c8048df144d39157f2918473e43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:04.935695 9921 tablet_bootstrap.cc:492] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:04.935943 9920 tablet_bootstrap.cc:492] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:04.937281 9921 tablet_bootstrap.cc:654] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:04.937402 9920 tablet_bootstrap.cc:654] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:04.937880 9921 log.cc:824] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:04.937970 9920 log.cc:824] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:04.939630 9920 tablet_bootstrap.cc:492] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:04.939630 9921 tablet_bootstrap.cc:492] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:04.939814 9921 ts_tablet_manager.cc:1367] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:04.939810 9920 ts_tablet_manager.cc:1367] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:04.943578 9921 raft_consensus.cc:348] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.943578 9920 raft_consensus.cc:348] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:04.943809 9920 raft_consensus.cc:374] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:04.943809 9921 raft_consensus.cc:374] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:04.943884 9921 raft_consensus.cc:729] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:04.943885 9920 raft_consensus.cc:729] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:04.944361 9921 consensus_queue.cc:260] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.944361 9920 consensus_queue.cc:260] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:04.944517 9920 raft_consensus.cc:388] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:04.944518 9921 raft_consensus.cc:388] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:04.944593 9921 raft_consensus.cc:482] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:04.944593 9920 raft_consensus.cc:482] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:04.944718 9920 raft_consensus.cc:3037] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:04.944712 9921 raft_consensus.cc:3037] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:04.945748 9921 raft_consensus.cc:504] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.945748 9920 raft_consensus.cc:504] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:04.946254 9920 leader_election.cc:302] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:04.946254 9921 leader_election.cc:302] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:04.946624 9921 leader_election.cc:288] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:04.946624 9920 leader_election.cc:288] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:04.946774 9925 raft_consensus.cc:2781] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:04.947016 9927 raft_consensus.cc:2781] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:04.947016 9925 raft_consensus.cc:686] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:04.947330 9925 consensus_queue.cc:237] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.947502 9920 ts_tablet_manager.cc:1398] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 14:09:04.947681 9927 raft_consensus.cc:686] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:04.947914 9921 ts_tablet_manager.cc:1398] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20240501 14:09:04.948024 9927 consensus_queue.cc:237] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:04.948114 9921 tablet_bootstrap.cc:492] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:04.950053 9921 tablet_bootstrap.cc:654] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:04.950103 8909 catalog_manager.cc:5484] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:04.950665 8910 catalog_manager.cc:5484] T dfb5160c34df401fb54fb75eab797b92 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:04.951344 9921 tablet_bootstrap.cc:492] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:04.951431 9921 ts_tablet_manager.cc:1367] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:04.951990 9921 raft_consensus.cc:348] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.952235 9921 raft_consensus.cc:374] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:04.952291 9921 raft_consensus.cc:729] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:04.952442 9921 consensus_queue.cc:260] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.952543 9921 raft_consensus.cc:388] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:04.952576 9921 raft_consensus.cc:482] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:04.952667 9921 raft_consensus.cc:3037] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:04.953548 9921 raft_consensus.cc:504] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.953655 9921 leader_election.cc:302] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:04.953747 9921 leader_election.cc:288] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:04.953810 9928 raft_consensus.cc:2781] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:04.953898 9928 raft_consensus.cc:686] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:04.953907 9921 ts_tablet_manager.cc:1398] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:04.954038 9928 consensus_queue.cc:237] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:04.955735 8910 catalog_manager.cc:5484] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:05.070426 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.070695 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.073670 9936 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 63.8771 ms
time per row: 0.0319386 ms
I20240501 14:09:05.176715 8902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59986:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:05.181722 9163 tablet_service.cc:1467] Processing CreateTablet for tablet a7254f1aa0cd41e1bf8226d31c173095 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d838f67b9a3341e8a0d8ba0a99c0001c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.181722 9162 tablet_service.cc:1467] Processing CreateTablet for tablet ee739a5adf394e05b951d1e9c3ae5abe (DEFAULT_TABLE table=rebalance_test_table_1 [id=d838f67b9a3341e8a0d8ba0a99c0001c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.181867 9032 tablet_service.cc:1467] Processing CreateTablet for tablet 04fc0a9f61d2446c868c2bc2a7792da7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d838f67b9a3341e8a0d8ba0a99c0001c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.182061 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7254f1aa0cd41e1bf8226d31c173095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.182094 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04fc0a9f61d2446c868c2bc2a7792da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.182238 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee739a5adf394e05b951d1e9c3ae5abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.183770 9920 tablet_bootstrap.cc:492] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:05.184077 9921 tablet_bootstrap.cc:492] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:05.184655 9920 tablet_bootstrap.cc:654] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.184739 9921 tablet_bootstrap.cc:654] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.185664 9921 tablet_bootstrap.cc:492] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:05.185739 9921 ts_tablet_manager.cc:1367] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.185781 9920 tablet_bootstrap.cc:492] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:05.185861 9920 ts_tablet_manager.cc:1367] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.186213 9921 raft_consensus.cc:348] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.186300 9921 raft_consensus.cc:374] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.186334 9921 raft_consensus.cc:729] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.186328 9920 raft_consensus.cc:348] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.186444 9920 raft_consensus.cc:374] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.186471 9920 raft_consensus.cc:729] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.186486 9921 consensus_queue.cc:260] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.186571 9921 raft_consensus.cc:388] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.186575 9920 consensus_queue.cc:260] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.186614 9921 raft_consensus.cc:482] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.186663 9921 raft_consensus.cc:3037] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.186666 9920 raft_consensus.cc:388] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.186700 9920 raft_consensus.cc:482] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.186790 9920 raft_consensus.cc:3037] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.187300 9921 raft_consensus.cc:504] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.187348 9920 raft_consensus.cc:504] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.187434 9921 leader_election.cc:302] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:05.187474 9920 leader_election.cc:302] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:05.187533 9921 leader_election.cc:288] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.187582 9925 raft_consensus.cc:2781] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.187682 9925 raft_consensus.cc:686] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:05.187755 9921 ts_tablet_manager.cc:1398] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:05.187790 9925 consensus_queue.cc:237] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.188705 9920 leader_election.cc:288] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.188778 9924 raft_consensus.cc:2781] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.188871 9924 raft_consensus.cc:686] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:05.188971 9924 consensus_queue.cc:237] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.189028 9920 ts_tablet_manager.cc:1398] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:05.189250 9920 tablet_bootstrap.cc:492] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:05.189217 8902 catalog_manager.cc:5484] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.190009 9920 tablet_bootstrap.cc:654] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.190568 8903 catalog_manager.cc:5484] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.191224 9920 tablet_bootstrap.cc:492] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:05.191300 9920 ts_tablet_manager.cc:1367] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:05.191682 9920 raft_consensus.cc:348] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.191747 9920 raft_consensus.cc:374] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.191776 9920 raft_consensus.cc:729] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.191885 9920 consensus_queue.cc:260] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.191962 9920 raft_consensus.cc:388] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.191998 9920 raft_consensus.cc:482] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.192036 9920 raft_consensus.cc:3037] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.192602 9920 raft_consensus.cc:504] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.192723 9920 leader_election.cc:302] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:05.192816 9920 leader_election.cc:288] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.192865 9924 raft_consensus.cc:2781] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.192958 9924 raft_consensus.cc:686] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:05.193017 9920 ts_tablet_manager.cc:1398] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:05.193078 9924 consensus_queue.cc:237] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.195353 8903 catalog_manager.cc:5484] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:05.302582 9978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.302832 9978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.305637 9978 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 61.7905 ms
time per row: 0.0308953 ms
I20240501 14:09:05.406103 8902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59986:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:05.411401 9032 tablet_service.cc:1467] Processing CreateTablet for tablet 4472ad76c4fa46dc94e1824b2972a43d (DEFAULT_TABLE table=rebalance_test_table_2 [id=32df158a725c4ec1883752d3033e823c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.411470 9162 tablet_service.cc:1467] Processing CreateTablet for tablet 1202b0b9e74e4f4bbe7a31afe4fbf553 (DEFAULT_TABLE table=rebalance_test_table_2 [id=32df158a725c4ec1883752d3033e823c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.411761 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4472ad76c4fa46dc94e1824b2972a43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.411782 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1202b0b9e74e4f4bbe7a31afe4fbf553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.412086 9031 tablet_service.cc:1467] Processing CreateTablet for tablet d1d8e1ca84bc4b5980dc5a47db9412bf (DEFAULT_TABLE table=rebalance_test_table_2 [id=32df158a725c4ec1883752d3033e823c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.412287 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d8e1ca84bc4b5980dc5a47db9412bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.413341 9921 tablet_bootstrap.cc:492] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:05.413786 9920 tablet_bootstrap.cc:492] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:05.414137 9921 tablet_bootstrap.cc:654] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.414705 9920 tablet_bootstrap.cc:654] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.415287 9921 tablet_bootstrap.cc:492] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:05.415355 9921 ts_tablet_manager.cc:1367] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.415807 9920 tablet_bootstrap.cc:492] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:05.415894 9920 ts_tablet_manager.cc:1367] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:05.415865 9921 raft_consensus.cc:348] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.415954 9921 raft_consensus.cc:374] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.415999 9921 raft_consensus.cc:729] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.416129 9921 consensus_queue.cc:260] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.416222 9921 raft_consensus.cc:388] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.416275 9921 raft_consensus.cc:482] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.416322 9921 raft_consensus.cc:3037] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.416374 9920 raft_consensus.cc:348] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.416471 9920 raft_consensus.cc:374] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.416508 9920 raft_consensus.cc:729] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.416616 9920 consensus_queue.cc:260] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.416694 9920 raft_consensus.cc:388] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.416724 9920 raft_consensus.cc:482] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.416760 9920 raft_consensus.cc:3037] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.417212 9921 raft_consensus.cc:504] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.417340 9921 leader_election.cc:302] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:05.417464 9921 leader_election.cc:288] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.417522 9925 raft_consensus.cc:2781] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.417614 9925 raft_consensus.cc:686] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:05.417737 9921 ts_tablet_manager.cc:1398] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:05.417726 9925 consensus_queue.cc:237] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.417611 9920 raft_consensus.cc:504] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.417889 9921 tablet_bootstrap.cc:492] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:05.417953 9920 leader_election.cc:302] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:05.418076 9920 leader_election.cc:288] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.418128 9927 raft_consensus.cc:2781] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.418220 9927 raft_consensus.cc:686] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:05.418347 9927 consensus_queue.cc:237] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.418673 9920 ts_tablet_manager.cc:1398] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:05.419754 8903 catalog_manager.cc:5484] T 1202b0b9e74e4f4bbe7a31afe4fbf553 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.419797 8902 catalog_manager.cc:5484] T 4472ad76c4fa46dc94e1824b2972a43d P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.420485 9921 tablet_bootstrap.cc:654] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.422008 9921 tablet_bootstrap.cc:492] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:05.422353 9921 ts_tablet_manager.cc:1367] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:05.422955 9921 raft_consensus.cc:348] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.423045 9921 raft_consensus.cc:374] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.423074 9921 raft_consensus.cc:729] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.423180 9921 consensus_queue.cc:260] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.423264 9921 raft_consensus.cc:388] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.423300 9921 raft_consensus.cc:482] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.423336 9921 raft_consensus.cc:3037] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.424127 9921 raft_consensus.cc:504] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.424252 9921 leader_election.cc:302] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:05.424361 9921 leader_election.cc:288] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.424403 9928 raft_consensus.cc:2781] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.424499 9928 raft_consensus.cc:686] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:05.424693 9921 ts_tablet_manager.cc:1398] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:05.424633 9928 consensus_queue.cc:237] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.425834 8902 catalog_manager.cc:5484] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:05.534919 10012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.535141 10012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.538043 10012 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 66.8875 ms
time per row: 0.0334438 ms
I20240501 14:09:05.643472 8903 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59986:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:05.648736 9162 tablet_service.cc:1467] Processing CreateTablet for tablet a88e323853ac475a853775f585b71cc4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3f15950b101c40918ad69275e08ab5af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.648815 9031 tablet_service.cc:1467] Processing CreateTablet for tablet df4790f45f44444bb45df0877d5910a2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3f15950b101c40918ad69275e08ab5af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.648808 9163 tablet_service.cc:1467] Processing CreateTablet for tablet 94ed7526967d4cafaa50839b73861a76 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3f15950b101c40918ad69275e08ab5af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.649029 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88e323853ac475a853775f585b71cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.649076 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4790f45f44444bb45df0877d5910a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.649164 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ed7526967d4cafaa50839b73861a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.650664 9921 tablet_bootstrap.cc:492] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:05.651124 9920 tablet_bootstrap.cc:492] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:05.651510 9921 tablet_bootstrap.cc:654] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.651824 9920 tablet_bootstrap.cc:654] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.652554 9921 tablet_bootstrap.cc:492] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:05.652632 9921 ts_tablet_manager.cc:1367] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.652856 9920 tablet_bootstrap.cc:492] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:05.652953 9920 ts_tablet_manager.cc:1367] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.653035 9921 raft_consensus.cc:348] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.653117 9921 raft_consensus.cc:374] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.653136 9921 raft_consensus.cc:729] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.653308 9921 consensus_queue.cc:260] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.653348 9920 raft_consensus.cc:348] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.653398 9921 raft_consensus.cc:388] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.653441 9921 raft_consensus.cc:482] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.653442 9920 raft_consensus.cc:374] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.653474 9920 raft_consensus.cc:729] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.653487 9921 raft_consensus.cc:3037] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.653582 9920 consensus_queue.cc:260] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.653647 9920 raft_consensus.cc:388] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.653666 9920 raft_consensus.cc:482] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.653697 9920 raft_consensus.cc:3037] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.654214 9921 raft_consensus.cc:504] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.654338 9921 leader_election.cc:302] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:05.654395 9920 raft_consensus.cc:504] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.654441 9921 leader_election.cc:288] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.654520 9920 leader_election.cc:302] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:05.654572 9926 raft_consensus.cc:2781] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.654644 9926 raft_consensus.cc:686] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:05.654800 9920 leader_election.cc:288] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.654831 9927 raft_consensus.cc:2781] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.654918 9927 raft_consensus.cc:686] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:05.655030 9927 consensus_queue.cc:237] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.654781 9926 consensus_queue.cc:237] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.655128 9920 ts_tablet_manager.cc:1398] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:05.655251 9920 tablet_bootstrap.cc:492] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:05.654675 9921 ts_tablet_manager.cc:1398] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:05.656082 9920 tablet_bootstrap.cc:654] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.656633 8903 catalog_manager.cc:5484] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.656631 8904 catalog_manager.cc:5484] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.657089 9920 tablet_bootstrap.cc:492] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:05.657151 9920 ts_tablet_manager.cc:1367] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:05.657524 9920 raft_consensus.cc:348] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.657608 9920 raft_consensus.cc:374] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.657650 9920 raft_consensus.cc:729] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.657768 9920 consensus_queue.cc:260] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.657845 9920 raft_consensus.cc:388] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.657871 9920 raft_consensus.cc:482] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.657908 9920 raft_consensus.cc:3037] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.658757 9920 raft_consensus.cc:504] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.658881 9920 leader_election.cc:302] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:05.658962 9920 leader_election.cc:288] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.659147 9920 ts_tablet_manager.cc:1398] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:05.659286 9929 raft_consensus.cc:2781] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.659389 9929 raft_consensus.cc:686] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:05.659498 9929 consensus_queue.cc:237] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.660848 8903 catalog_manager.cc:5484] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:05.768014 10043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:05.768273 10043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:05.771255 10043 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 58.1222 ms
time per row: 0.0290611 ms
I20240501 14:09:05.873891 8904 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59986:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:05.879329 9031 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4aea6894f74532b1527bad663d9288 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cbae0c7e67394a93876876fc67386fbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.879401 9032 tablet_service.cc:1467] Processing CreateTablet for tablet 73a00189fb5d44f9b67639e8b8ed3303 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cbae0c7e67394a93876876fc67386fbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.879377 9163 tablet_service.cc:1467] Processing CreateTablet for tablet 06a2cf8244224085bccaf81a1c6cf85d (DEFAULT_TABLE table=rebalance_test_table_4 [id=cbae0c7e67394a93876876fc67386fbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:05.879673 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a00189fb5d44f9b67639e8b8ed3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.879688 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a2cf8244224085bccaf81a1c6cf85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.879961 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4aea6894f74532b1527bad663d9288. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:05.881731 9921 tablet_bootstrap.cc:492] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:05.881804 9920 tablet_bootstrap.cc:492] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Bootstrap starting.
I20240501 14:09:05.882586 9921 tablet_bootstrap.cc:654] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.882608 9920 tablet_bootstrap.cc:654] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.883620 9921 tablet_bootstrap.cc:492] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:05.883663 9920 tablet_bootstrap.cc:492] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: No bootstrap required, opened a new log
I20240501 14:09:05.883713 9921 ts_tablet_manager.cc:1367] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:05.883726 9920 ts_tablet_manager.cc:1367] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:05.884074 9921 raft_consensus.cc:348] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.884075 9920 raft_consensus.cc:348] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.884163 9920 raft_consensus.cc:374] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.884167 9921 raft_consensus.cc:374] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.884186 9920 raft_consensus.cc:729] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.884205 9921 raft_consensus.cc:729] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.884300 9920 consensus_queue.cc:260] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.884382 9920 raft_consensus.cc:388] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.884348 9921 consensus_queue.cc:260] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.884433 9920 raft_consensus.cc:482] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.884445 9921 raft_consensus.cc:388] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.884486 9920 raft_consensus.cc:3037] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.884497 9921 raft_consensus.cc:482] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.884546 9921 raft_consensus.cc:3037] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.885285 9920 raft_consensus.cc:504] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.885282 9921 raft_consensus.cc:504] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.885409 9921 leader_election.cc:302] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:05.885412 9920 leader_election.cc:302] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:05.885514 9921 leader_election.cc:288] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.885562 9926 raft_consensus.cc:2781] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.885668 9926 raft_consensus.cc:686] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:05.885821 9921 ts_tablet_manager.cc:1398] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:05.885790 9926 consensus_queue.cc:237] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.885939 9927 raft_consensus.cc:2781] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.885967 9921 tablet_bootstrap.cc:492] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Bootstrap starting.
I20240501 14:09:05.886019 9927 raft_consensus.cc:686] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Leader. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:05.886147 9927 consensus_queue.cc:237] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } }
I20240501 14:09:05.886850 9921 tablet_bootstrap.cc:654] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:05.887423 8904 catalog_manager.cc:5484] T 7a4aea6894f74532b1527bad663d9288 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.887742 8903 catalog_manager.cc:5484] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2). New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:05.888314 9921 tablet_bootstrap.cc:492] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: No bootstrap required, opened a new log
I20240501 14:09:05.888393 9921 ts_tablet_manager.cc:1367] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:05.888803 9921 raft_consensus.cc:348] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.888883 9921 raft_consensus.cc:374] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:05.888917 9921 raft_consensus.cc:729] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Initialized, Role: FOLLOWER
I20240501 14:09:05.889042 9921 consensus_queue.cc:260] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.889117 9921 raft_consensus.cc:388] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:05.889173 9921 raft_consensus.cc:482] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:05.889217 9921 raft_consensus.cc:3037] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:05.889321 9920 leader_election.cc:288] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.889618 9920 ts_tablet_manager.cc:1398] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:05.890085 9921 raft_consensus.cc:504] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.890197 9921 leader_election.cc:302] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:05.890290 9921 leader_election.cc:288] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:05.890348 9926 raft_consensus.cc:2781] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:05.890444 9926 raft_consensus.cc:686] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Leader. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:05.890491 9921 ts_tablet_manager.cc:1398] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:05.890537 9926 consensus_queue.cc:237] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } }
I20240501 14:09:05.892066 8904 catalog_manager.cc:5484] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1). New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:06.000020 10076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.000273 10076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.003334 10076 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 73.6934 ms
time per row: 0.0368467 ms
I20240501 14:09:06.117149 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.3:37685
--local_ip_for_outbound_sockets=127.6.82.3
--tserver_master_addrs=127.6.82.62:37407
--webserver_port=34025
--webserver_interface=127.6.82.3
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:06.223992 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.224224 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:06.224268 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.227418 10101 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:06.227485 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:06.227566 10101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.3
I20240501 14:09:06.231714 10101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.3:37685
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.82.3
--webserver_port=34025
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.10101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.3
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:06.232709 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:06.233858 10101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:06.241361 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.241425 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.241590 10101 server_base.cc:1034] running on GCE node
W20240501 14:09:06.241356 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.242090 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:06.242646 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:06.243805 10101 hybrid_clock.cc:648] HybridClock initialized: now 1714572546243798 us; error 16 us; skew 500 ppm
I20240501 14:09:06.245600 10101 webserver.cc:460] Webserver started at http://127.6.82.3:34025/ using document root <none> and password file <none>
I20240501 14:09:06.246111 10101 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:06.246173 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:06.249390 10101 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.251233 10115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.252166 10101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.252260 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f2040b402c764007a4b1d969db917ede"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:06.252637 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:06.293663 10101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:06.294333 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:06.294479 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:06.295009 10101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:06.296327 10122 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 14:09:06.300122 10101 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:09:06.300204 10101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:06.300288 10101 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:09:06.301793 10101 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:09:06.301841 10101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:06.302112 10122 tablet_bootstrap.cc:492] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Bootstrap starting.
I20240501 14:09:06.311793 10101 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.3:37685
I20240501 14:09:06.311816 10229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.3:37685 every 8 connection(s)
I20240501 14:09:06.312848 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:06.313297 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 10101
I20240501 14:09:06.313694 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.4:33949
--local_ip_for_outbound_sockets=127.6.82.4
--tserver_master_addrs=127.6.82.62:37407
--webserver_port=44769
--webserver_interface=127.6.82.4
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:06.316907 10122 log.cc:824] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:06.319905 10122 tablet_bootstrap.cc:492] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:06.320341 10122 tablet_bootstrap.cc:492] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Bootstrap complete.
I20240501 14:09:06.321730 10122 ts_tablet_manager.cc:1367] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20240501 14:09:06.321983 10230 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:06.322204 10230 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:06.322713 10230 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:06.324447 8904 ts_manager.cc:194] Re-registered known tserver with Master: f2040b402c764007a4b1d969db917ede (127.6.82.3:37685)
I20240501 14:09:06.325251 8904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.3:42671
I20240501 14:09:06.325794 10122 raft_consensus.cc:348] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:06.326232 10122 raft_consensus.cc:729] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2040b402c764007a4b1d969db917ede, State: Initialized, Role: FOLLOWER
I20240501 14:09:06.326802 10122 consensus_queue.cc:260] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:06.326959 10122 raft_consensus.cc:388] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:06.327041 10122 raft_consensus.cc:482] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:06.327173 10122 raft_consensus.cc:3037] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:06.329391 10122 raft_consensus.cc:504] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:06.329759 10122 leader_election.cc:302] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2040b402c764007a4b1d969db917ede; no voters:
I20240501 14:09:06.330109 10122 leader_election.cc:288] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 14:09:06.330241 10235 raft_consensus.cc:2781] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:06.330459 10235 raft_consensus.cc:686] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [term 2 LEADER]: Becoming Leader. State: Replica: f2040b402c764007a4b1d969db917ede, State: Running, Role: LEADER
I20240501 14:09:06.330816 10230 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:06.330780 10235 consensus_queue.cc:237] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } }
I20240501 14:09:06.331023 10122 ts_tablet_manager.cc:1398] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede: Time spent starting tablet: real 0.009s user 0.012s sys 0.000s
I20240501 14:09:06.332947 8904 catalog_manager.cc:5484] T 6a41ee08ad7141f5bb360dc992a3c072 P f2040b402c764007a4b1d969db917ede reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "f2040b402c764007a4b1d969db917ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:06.431591 10232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.431906 10232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:06.432003 10232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.435367 10232 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:06.435436 10232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:06.435525 10232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.4
I20240501 14:09:06.439191 10232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.4:33949
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.82.4
--webserver_port=44769
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.10232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.4
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:06.440140 10232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:06.441303 10232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:06.448982 10247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.448990 10245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.448992 10244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.449303 10232 server_base.cc:1034] running on GCE node
I20240501 14:09:06.449724 10232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:06.450304 10232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:06.451481 10232 hybrid_clock.cc:648] HybridClock initialized: now 1714572546451469 us; error 28 us; skew 500 ppm
I20240501 14:09:06.453457 10232 webserver.cc:460] Webserver started at http://127.6.82.4:44769/ using document root <none> and password file <none>
I20240501 14:09:06.454123 10232 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:06.454211 10232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:06.457796 10232 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.459735 10253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.460778 10232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:06.460893 10232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fa75a3f967ef4d259a6e56f2b457b39b"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:06.461413 10232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:06.477558 10232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:06.478189 10232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:06.478355 10232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:06.478930 10232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:06.479804 10232 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:06.479851 10232 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.479890 10232 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:06.479912 10232 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.489018 10232 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.4:33949
I20240501 14:09:06.489050 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.4:33949 every 8 connection(s)
I20240501 14:09:06.490049 10232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:06.491761 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 10232
I20240501 14:09:06.492158 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.5:46807
--local_ip_for_outbound_sockets=127.6.82.5
--tserver_master_addrs=127.6.82.62:37407
--webserver_port=36645
--webserver_interface=127.6.82.5
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:06.499589 10367 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:06.499830 10367 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:06.500413 10367 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:06.501542 8904 ts_manager.cc:194] Re-registered known tserver with Master: fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4:33949)
I20240501 14:09:06.502097 8904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.4:34403
W20240501 14:09:06.600924 10370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.601174 10370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:06.601245 10370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.604310 10370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:06.604388 10370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:06.604480 10370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.5
I20240501 14:09:06.608078 10370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.5:46807
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.82.5
--webserver_port=36645
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.10370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.5
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:06.609009 10370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:06.610121 10370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:06.617878 10376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.617889 10378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.617924 10375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.618121 10370 server_base.cc:1034] running on GCE node
I20240501 14:09:06.618546 10370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:06.619117 10370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:06.620265 10370 hybrid_clock.cc:648] HybridClock initialized: now 1714572546620257 us; error 25 us; skew 500 ppm
I20240501 14:09:06.622035 10370 webserver.cc:460] Webserver started at http://127.6.82.5:36645/ using document root <none> and password file <none>
I20240501 14:09:06.622570 10370 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:06.622655 10370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:06.625926 10370 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.627880 10384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.628919 10370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:06.629033 10370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b42390c48f8c4e71a476d98115a685ac"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:06.629442 10370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:06.650297 10370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:06.651017 10370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:06.651180 10370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:06.651721 10370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:06.652592 10370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:06.652640 10370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.652701 10370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:06.652729 10370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.661547 10370 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.5:46807
I20240501 14:09:06.661576 10497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.5:46807 every 8 connection(s)
I20240501 14:09:06.662562 10370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:06.668361 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 10370
I20240501 14:09:06.668745 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.6:34217
--local_ip_for_outbound_sockets=127.6.82.6
--tserver_master_addrs=127.6.82.62:37407
--webserver_port=36697
--webserver_interface=127.6.82.6
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:06.671303 10498 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:06.671519 10498 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:06.672065 10498 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:06.673074 8904 ts_manager.cc:194] Re-registered known tserver with Master: b42390c48f8c4e71a476d98115a685ac (127.6.82.5:46807)
I20240501 14:09:06.673599 8904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.5:37357
W20240501 14:09:06.776329 10501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.776566 10501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:06.776643 10501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.779673 10501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:06.779748 10501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:06.779839 10501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.6
I20240501 14:09:06.783330 10501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.6:34217
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.82.6
--webserver_port=36697
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.10501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.6
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:06.784222 10501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:06.785326 10501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:06.792830 10506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.792856 10509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.792861 10507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.793047 10501 server_base.cc:1034] running on GCE node
I20240501 14:09:06.793448 10501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:06.793958 10501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:06.795094 10501 hybrid_clock.cc:648] HybridClock initialized: now 1714572546795068 us; error 45 us; skew 500 ppm
I20240501 14:09:06.796840 10501 webserver.cc:460] Webserver started at http://127.6.82.6:36697/ using document root <none> and password file <none>
I20240501 14:09:06.797338 10501 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:06.797402 10501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:06.800607 10501 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:06.802378 10515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.803386 10501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.803475 10501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e32ae6129e5a47aa955c480ff1962e06"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:06.803870 10501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:06.834687 10501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:06.835381 10501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:06.835543 10501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:06.836078 10501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:06.836975 10501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:06.837025 10501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.837085 10501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:06.837113 10501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.845536 10501 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.6:34217
I20240501 14:09:06.845556 10628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.6:34217 every 8 connection(s)
I20240501 14:09:06.846529 10501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:06.854840 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 10501
I20240501 14:09:06.854897 10629 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:06.855098 10629 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:06.855243 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.7:42267
--local_ip_for_outbound_sockets=127.6.82.7
--tserver_master_addrs=127.6.82.62:37407
--webserver_port=40989
--webserver_interface=127.6.82.7
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:06.855687 10629 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:06.856766 8904 ts_manager.cc:194] Re-registered known tserver with Master: e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6:34217)
I20240501 14:09:06.857347 8904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.6:50423
W20240501 14:09:06.963246 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:06.963454 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:06.963519 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:06.966840 10632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:06.966925 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:06.967046 10632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.7
I20240501 14:09:06.970824 10632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.7:42267
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.82.7
--webserver_port=40989
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.10632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.7
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:06.971729 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:06.972890 10632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:06.980576 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:06.980597 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.980778 10632 server_base.cc:1034] running on GCE node
W20240501 14:09:06.980576 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:06.981241 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:06.981748 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:06.982916 10632 hybrid_clock.cc:648] HybridClock initialized: now 1714572546982906 us; error 27 us; skew 500 ppm
I20240501 14:09:06.984634 10632 webserver.cc:460] Webserver started at http://127.6.82.7:40989/ using document root <none> and password file <none>
I20240501 14:09:06.985123 10632 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:06.985193 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:06.988320 10632 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:06.990058 10646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:06.991096 10632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:06.991196 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "06295523e9de4e2dae4b0909b0a04ea6"
format_stamp: "Formatted at 2024-05-01 14:09:00 on dist-test-slave-crgf"
I20240501 14:09:06.991582 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:07.005780 10632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:07.006346 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:07.006476 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:07.007009 10632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:07.007843 10632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:07.007895 10632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.007944 10632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:07.007975 10632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:07.016098 10632 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.7:42267
I20240501 14:09:07.016186 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.7:42267 every 8 connection(s)
I20240501 14:09:07.017133 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:07.021059 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 10632
I20240501 14:09:07.025683 10760 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:07.025899 10760 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:07.026376 10760 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:07.027329 8904 ts_manager.cc:194] Re-registered known tserver with Master: 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7:42267)
I20240501 14:09:07.028000 8904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.7:37835
I20240501 14:09:07.205549 9163 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:07.209895 10694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.210173 10563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.210836 10432 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.215720 9031 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:07.219270 10301 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.220257 10153 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.346798 9163 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:09:07.347375 10694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.349820 9031 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:07.356804 10563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.357141 10301 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.366039 10153 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:07.380787 10432 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:07.408545 9183 consensus_queue.cc:237] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } }
I20240501 14:09:07.409927 10850 raft_consensus.cc:2932] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } } }
I20240501 14:09:07.413146 8903 catalog_manager.cc:5484] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index -1 to 4, NON_VOTER 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.414538 9119 consensus_peers.cc:485] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a -> Peer 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7:42267): Couldn't send request to peer 06295523e9de4e2dae4b0909b0a04ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a88e323853ac475a853775f585b71cc4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.420956 9052 consensus_queue.cc:237] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } }
I20240501 14:09:07.422287 10864 raft_consensus.cc:2932] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } } }
I20240501 14:09:07.425055 8909 catalog_manager.cc:5484] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index -1 to 4, NON_VOTER e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.427487 8988 consensus_peers.cc:485] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af -> Peer e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6:34217): Couldn't send request to peer e32ae6129e5a47aa955c480ff1962e06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73a00189fb5d44f9b67639e8b8ed3303. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.431748 9183 consensus_queue.cc:237] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } }
I20240501 14:09:07.432680 10850 raft_consensus.cc:2932] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b42390c48f8c4e71a476d98115a685ac (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } } }
I20240501 14:09:07.436605 8909 catalog_manager.cc:5484] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index -1 to 4, NON_VOTER b42390c48f8c4e71a476d98115a685ac (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.437289 9118 consensus_peers.cc:485] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a -> Peer b42390c48f8c4e71a476d98115a685ac (127.6.82.5:46807): Couldn't send request to peer b42390c48f8c4e71a476d98115a685ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee739a5adf394e05b951d1e9c3ae5abe. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.442816 9052 consensus_queue.cc:237] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } }
I20240501 14:09:07.443910 10863 raft_consensus.cc:2932] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } } }
I20240501 14:09:07.446271 8903 catalog_manager.cc:5484] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index -1 to 4, NON_VOTER fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.447839 8986 consensus_peers.cc:485] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af -> Peer fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4:33949): Couldn't send request to peer fa75a3f967ef4d259a6e56f2b457b39b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e138c8048df144d39157f2918473e43d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.453148 9052 consensus_queue.cc:237] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: true } }
I20240501 14:09:07.454064 10863 raft_consensus.cc:2932] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f2040b402c764007a4b1d969db917ede (127.6.82.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: true } } }
I20240501 14:09:07.457084 8903 catalog_manager.cc:5484] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index -1 to 4, NON_VOTER f2040b402c764007a4b1d969db917ede (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.458667 8986 consensus_peers.cc:485] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af -> Peer f2040b402c764007a4b1d969db917ede (127.6.82.3:37685): Couldn't send request to peer f2040b402c764007a4b1d969db917ede. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7316a1b44e054c96acc41b645b2d33c4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.463593 9183 consensus_queue.cc:237] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } }
I20240501 14:09:07.464690 10851 raft_consensus.cc:2932] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } } }
W20240501 14:09:07.465013 9119 consensus_peers.cc:485] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a -> Peer 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7:42267): Couldn't send request to peer 06295523e9de4e2dae4b0909b0a04ea6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06a2cf8244224085bccaf81a1c6cf85d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.467036 8903 catalog_manager.cc:5484] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index -1 to 4, NON_VOTER 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:07.472980 9183 consensus_queue.cc:237] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } }
I20240501 14:09:07.473805 10890 raft_consensus.cc:2932] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } } }
I20240501 14:09:07.476053 8903 catalog_manager.cc:5484] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index -1 to 4, NON_VOTER e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.477260 9119 consensus_peers.cc:485] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a -> Peer e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6:34217): Couldn't send request to peer e32ae6129e5a47aa955c480ff1962e06. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7254f1aa0cd41e1bf8226d31c173095. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.482618 9052 consensus_queue.cc:237] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } }
I20240501 14:09:07.483549 10863 raft_consensus.cc:2932] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b42390c48f8c4e71a476d98115a685ac (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } } }
I20240501 14:09:07.485958 8903 catalog_manager.cc:5484] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index -1 to 4, NON_VOTER b42390c48f8c4e71a476d98115a685ac (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:07.487434 8987 consensus_peers.cc:485] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af -> Peer b42390c48f8c4e71a476d98115a685ac (127.6.82.5:46807): Couldn't send request to peer b42390c48f8c4e71a476d98115a685ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1d8e1ca84bc4b5980dc5a47db9412bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.492606 9052 consensus_queue.cc:237] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } }
I20240501 14:09:07.493578 10864 raft_consensus.cc:2932] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } } }
W20240501 14:09:07.493849 8986 consensus_peers.cc:485] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af -> Peer fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4:33949): Couldn't send request to peer fa75a3f967ef4d259a6e56f2b457b39b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df4790f45f44444bb45df0877d5910a2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:07.495875 8903 catalog_manager.cc:5484] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index -1 to 4, NON_VOTER fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:07.504063 10367 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:07.675779 10498 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:07.800354 10999 ts_tablet_manager.cc:902] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Initiating tablet copy from peer b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:07.801084 10999 tablet_copy_client.cc:308] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:43063
I20240501 14:09:07.804724 9072 tablet_copy_service.cc:140] P b4c1a584bc6542f68b6d6912b091f1af: Received BeginTabletCopySession request for tablet 73a00189fb5d44f9b67639e8b8ed3303 from peer e32ae6129e5a47aa955c480ff1962e06 ({username='slave'} at 127.6.82.6:47155)
I20240501 14:09:07.804874 9072 tablet_copy_service.cc:161] P b4c1a584bc6542f68b6d6912b091f1af: Beginning new tablet copy session on tablet 73a00189fb5d44f9b67639e8b8ed3303 from peer e32ae6129e5a47aa955c480ff1962e06 at {username='slave'} at 127.6.82.6:47155: session id = e32ae6129e5a47aa955c480ff1962e06-73a00189fb5d44f9b67639e8b8ed3303
I20240501 14:09:07.805990 9072 tablet_copy_source_session.cc:205] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.809137 10999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a00189fb5d44f9b67639e8b8ed3303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.814020 10999 tablet_copy_client.cc:783] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.814195 10999 tablet_copy_client.cc:647] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.815837 10999 tablet_copy_client.cc:523] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.817313 10999 tablet_bootstrap.cc:492] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Bootstrap starting.
I20240501 14:09:07.822535 10999 log.cc:824] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:07.834695 11006 ts_tablet_manager.cc:902] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: Initiating tablet copy from peer b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:07.835412 11006 tablet_copy_client.cc:308] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:43063
I20240501 14:09:07.838927 9072 tablet_copy_service.cc:140] P b4c1a584bc6542f68b6d6912b091f1af: Received BeginTabletCopySession request for tablet 7316a1b44e054c96acc41b645b2d33c4 from peer f2040b402c764007a4b1d969db917ede ({username='slave'} at 127.6.82.3:38013)
I20240501 14:09:07.839048 9072 tablet_copy_service.cc:161] P b4c1a584bc6542f68b6d6912b091f1af: Beginning new tablet copy session on tablet 7316a1b44e054c96acc41b645b2d33c4 from peer f2040b402c764007a4b1d969db917ede at {username='slave'} at 127.6.82.3:38013: session id = f2040b402c764007a4b1d969db917ede-7316a1b44e054c96acc41b645b2d33c4
I20240501 14:09:07.840004 9072 tablet_copy_source_session.cc:205] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.841564 11006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7316a1b44e054c96acc41b645b2d33c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.842420 10999 tablet_bootstrap.cc:492] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:07.842803 10999 tablet_bootstrap.cc:492] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Bootstrap complete.
I20240501 14:09:07.843077 10999 ts_tablet_manager.cc:1367] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.005s
I20240501 14:09:07.844098 11006 tablet_copy_client.cc:783] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.844254 11006 tablet_copy_client.cc:647] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.845279 10999 raft_consensus.cc:348] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } }
I20240501 14:09:07.845547 10999 raft_consensus.cc:729] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e32ae6129e5a47aa955c480ff1962e06, State: Initialized, Role: LEARNER
I20240501 14:09:07.845727 11006 tablet_copy_client.cc:523] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.845906 10999 consensus_queue.cc:260] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } }
I20240501 14:09:07.846661 10999 ts_tablet_manager.cc:1398] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:07.847039 10629 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:07.847101 11006 tablet_bootstrap.cc:492] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: Bootstrap starting.
I20240501 14:09:07.847630 9072 tablet_copy_service.cc:342] P b4c1a584bc6542f68b6d6912b091f1af: Request end of tablet copy session e32ae6129e5a47aa955c480ff1962e06-73a00189fb5d44f9b67639e8b8ed3303 received from {username='slave'} at 127.6.82.6:47155
I20240501 14:09:07.847721 9072 tablet_copy_service.cc:433] P b4c1a584bc6542f68b6d6912b091f1af: ending tablet copy session e32ae6129e5a47aa955c480ff1962e06-73a00189fb5d44f9b67639e8b8ed3303 on tablet 73a00189fb5d44f9b67639e8b8ed3303 with peer e32ae6129e5a47aa955c480ff1962e06
I20240501 14:09:07.869874 11006 tablet_bootstrap.cc:492] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:07.870132 11006 tablet_bootstrap.cc:492] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: Bootstrap complete.
I20240501 14:09:07.870298 11006 ts_tablet_manager.cc:1367] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.004s
I20240501 14:09:07.870891 11006 raft_consensus.cc:348] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: true } }
I20240501 14:09:07.871002 11006 raft_consensus.cc:729] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2040b402c764007a4b1d969db917ede, State: Initialized, Role: LEARNER
I20240501 14:09:07.871098 11006 consensus_queue.cc:260] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: true } }
I20240501 14:09:07.871390 11006 ts_tablet_manager.cc:1398] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:07.872054 9072 tablet_copy_service.cc:342] P b4c1a584bc6542f68b6d6912b091f1af: Request end of tablet copy session f2040b402c764007a4b1d969db917ede-7316a1b44e054c96acc41b645b2d33c4 received from {username='slave'} at 127.6.82.3:38013
I20240501 14:09:07.872150 9072 tablet_copy_service.cc:433] P b4c1a584bc6542f68b6d6912b091f1af: ending tablet copy session f2040b402c764007a4b1d969db917ede-7316a1b44e054c96acc41b645b2d33c4 on tablet 7316a1b44e054c96acc41b645b2d33c4 with peer f2040b402c764007a4b1d969db917ede
I20240501 14:09:07.882470 11015 ts_tablet_manager.cc:902] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Initiating tablet copy from peer 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:07.883203 11015 tablet_copy_client.cc:308] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:37403
I20240501 14:09:07.886855 9203 tablet_copy_service.cc:140] P 90e73913f35c45dd8f2f517c3a385d3a: Received BeginTabletCopySession request for tablet a88e323853ac475a853775f585b71cc4 from peer 06295523e9de4e2dae4b0909b0a04ea6 ({username='slave'} at 127.6.82.7:36269)
I20240501 14:09:07.886982 9203 tablet_copy_service.cc:161] P 90e73913f35c45dd8f2f517c3a385d3a: Beginning new tablet copy session on tablet a88e323853ac475a853775f585b71cc4 from peer 06295523e9de4e2dae4b0909b0a04ea6 at {username='slave'} at 127.6.82.7:36269: session id = 06295523e9de4e2dae4b0909b0a04ea6-a88e323853ac475a853775f585b71cc4
I20240501 14:09:07.888069 9203 tablet_copy_source_session.cc:205] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.891256 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88e323853ac475a853775f585b71cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.896287 11015 tablet_copy_client.cc:783] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.896463 11015 tablet_copy_client.cc:647] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.897989 11015 tablet_copy_client.cc:523] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.899435 11015 tablet_bootstrap.cc:492] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Bootstrap starting.
I20240501 14:09:07.904304 11015 log.cc:824] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:07.923156 11015 tablet_bootstrap.cc:492] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:07.923463 11015 tablet_bootstrap.cc:492] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Bootstrap complete.
I20240501 14:09:07.923717 11015 ts_tablet_manager.cc:1367] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Time spent bootstrapping tablet: real 0.024s user 0.024s sys 0.000s
I20240501 14:09:07.925583 11015 raft_consensus.cc:348] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } }
I20240501 14:09:07.925776 11015 raft_consensus.cc:729] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06295523e9de4e2dae4b0909b0a04ea6, State: Initialized, Role: LEARNER
I20240501 14:09:07.926028 11015 consensus_queue.cc:260] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } }
I20240501 14:09:07.926597 10760 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:07.926684 11015 ts_tablet_manager.cc:1398] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:07.927471 9203 tablet_copy_service.cc:342] P 90e73913f35c45dd8f2f517c3a385d3a: Request end of tablet copy session 06295523e9de4e2dae4b0909b0a04ea6-a88e323853ac475a853775f585b71cc4 received from {username='slave'} at 127.6.82.7:36269
I20240501 14:09:07.927561 9203 tablet_copy_service.cc:433] P 90e73913f35c45dd8f2f517c3a385d3a: ending tablet copy session 06295523e9de4e2dae4b0909b0a04ea6-a88e323853ac475a853775f585b71cc4 on tablet a88e323853ac475a853775f585b71cc4 with peer 06295523e9de4e2dae4b0909b0a04ea6
I20240501 14:09:07.934221 11023 ts_tablet_manager.cc:902] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Initiating tablet copy from peer 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:07.934913 11023 tablet_copy_client.cc:308] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:37403
I20240501 14:09:07.937355 10999 ts_tablet_manager.cc:902] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: Initiating tablet copy from peer 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:07.937795 10999 tablet_copy_client.cc:308] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:37403
I20240501 14:09:07.938812 9203 tablet_copy_service.cc:140] P 90e73913f35c45dd8f2f517c3a385d3a: Received BeginTabletCopySession request for tablet ee739a5adf394e05b951d1e9c3ae5abe from peer b42390c48f8c4e71a476d98115a685ac ({username='slave'} at 127.6.82.5:38243)
I20240501 14:09:07.938900 9203 tablet_copy_service.cc:161] P 90e73913f35c45dd8f2f517c3a385d3a: Beginning new tablet copy session on tablet ee739a5adf394e05b951d1e9c3ae5abe from peer b42390c48f8c4e71a476d98115a685ac at {username='slave'} at 127.6.82.5:38243: session id = b42390c48f8c4e71a476d98115a685ac-ee739a5adf394e05b951d1e9c3ae5abe
I20240501 14:09:07.939599 9203 tablet_copy_source_session.cc:205] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.941064 9203 tablet_copy_service.cc:140] P 90e73913f35c45dd8f2f517c3a385d3a: Received BeginTabletCopySession request for tablet a7254f1aa0cd41e1bf8226d31c173095 from peer e32ae6129e5a47aa955c480ff1962e06 ({username='slave'} at 127.6.82.6:55725)
I20240501 14:09:07.941155 9203 tablet_copy_service.cc:161] P 90e73913f35c45dd8f2f517c3a385d3a: Beginning new tablet copy session on tablet a7254f1aa0cd41e1bf8226d31c173095 from peer e32ae6129e5a47aa955c480ff1962e06 at {username='slave'} at 127.6.82.6:55725: session id = e32ae6129e5a47aa955c480ff1962e06-a7254f1aa0cd41e1bf8226d31c173095
I20240501 14:09:07.941931 9203 tablet_copy_source_session.cc:205] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.942656 10999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7254f1aa0cd41e1bf8226d31c173095. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.942801 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee739a5adf394e05b951d1e9c3ae5abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.944307 10999 tablet_copy_client.cc:783] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.944439 10999 tablet_copy_client.cc:647] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.945407 10999 tablet_copy_client.cc:523] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.946465 10999 tablet_bootstrap.cc:492] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: Bootstrap starting.
I20240501 14:09:07.948578 11023 tablet_copy_client.cc:783] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.948770 11023 tablet_copy_client.cc:647] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.950219 11023 tablet_copy_client.cc:523] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.952018 11023 tablet_bootstrap.cc:492] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Bootstrap starting.
I20240501 14:09:07.958021 11023 log.cc:824] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:07.966024 11036 ts_tablet_manager.cc:902] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Initiating tablet copy from peer b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:07.966825 11036 tablet_copy_client.cc:308] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:43063
I20240501 14:09:07.968643 11015 ts_tablet_manager.cc:902] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: Initiating tablet copy from peer 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:07.969038 11015 tablet_copy_client.cc:308] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:37403
I20240501 14:09:07.969509 9203 tablet_copy_service.cc:140] P 90e73913f35c45dd8f2f517c3a385d3a: Received BeginTabletCopySession request for tablet 06a2cf8244224085bccaf81a1c6cf85d from peer 06295523e9de4e2dae4b0909b0a04ea6 ({username='slave'} at 127.6.82.7:36269)
I20240501 14:09:07.969610 9203 tablet_copy_service.cc:161] P 90e73913f35c45dd8f2f517c3a385d3a: Beginning new tablet copy session on tablet 06a2cf8244224085bccaf81a1c6cf85d from peer 06295523e9de4e2dae4b0909b0a04ea6 at {username='slave'} at 127.6.82.7:36269: session id = 06295523e9de4e2dae4b0909b0a04ea6-06a2cf8244224085bccaf81a1c6cf85d
I20240501 14:09:07.970476 9203 tablet_copy_source_session.cc:205] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.971374 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06a2cf8244224085bccaf81a1c6cf85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.972517 10999 tablet_bootstrap.cc:492] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:07.972796 10999 tablet_bootstrap.cc:492] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: Bootstrap complete.
I20240501 14:09:07.972965 10999 ts_tablet_manager.cc:1367] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240501 14:09:07.972986 9072 tablet_copy_service.cc:140] P b4c1a584bc6542f68b6d6912b091f1af: Received BeginTabletCopySession request for tablet e138c8048df144d39157f2918473e43d from peer fa75a3f967ef4d259a6e56f2b457b39b ({username='slave'} at 127.6.82.4:38267)
I20240501 14:09:07.973081 9072 tablet_copy_service.cc:161] P b4c1a584bc6542f68b6d6912b091f1af: Beginning new tablet copy session on tablet e138c8048df144d39157f2918473e43d from peer fa75a3f967ef4d259a6e56f2b457b39b at {username='slave'} at 127.6.82.4:38267: session id = fa75a3f967ef4d259a6e56f2b457b39b-e138c8048df144d39157f2918473e43d
I20240501 14:09:07.973521 11015 tablet_copy_client.cc:783] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.973546 10999 raft_consensus.cc:348] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } }
I20240501 14:09:07.973661 11015 tablet_copy_client.cc:647] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.973662 10999 raft_consensus.cc:729] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e32ae6129e5a47aa955c480ff1962e06, State: Initialized, Role: LEARNER
I20240501 14:09:07.973773 10999 consensus_queue.cc:260] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: true } }
I20240501 14:09:07.973903 9072 tablet_copy_source_session.cc:205] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:07.974020 10999 ts_tablet_manager.cc:1398] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:07.974872 9203 tablet_copy_service.cc:342] P 90e73913f35c45dd8f2f517c3a385d3a: Request end of tablet copy session e32ae6129e5a47aa955c480ff1962e06-a7254f1aa0cd41e1bf8226d31c173095 received from {username='slave'} at 127.6.82.6:55725
I20240501 14:09:07.974941 9203 tablet_copy_service.cc:433] P 90e73913f35c45dd8f2f517c3a385d3a: ending tablet copy session e32ae6129e5a47aa955c480ff1962e06-a7254f1aa0cd41e1bf8226d31c173095 on tablet a7254f1aa0cd41e1bf8226d31c173095 with peer e32ae6129e5a47aa955c480ff1962e06
I20240501 14:09:07.976171 11015 tablet_copy_client.cc:523] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.977448 11015 tablet_bootstrap.cc:492] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: Bootstrap starting.
I20240501 14:09:07.978715 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e138c8048df144d39157f2918473e43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:07.984902 11036 tablet_copy_client.cc:783] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:07.985071 11036 tablet_copy_client.cc:647] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:07.985422 11023 tablet_bootstrap.cc:492] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:07.985714 11023 tablet_bootstrap.cc:492] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Bootstrap complete.
I20240501 14:09:07.986014 11023 ts_tablet_manager.cc:1367] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.010s
I20240501 14:09:07.986904 11036 tablet_copy_client.cc:523] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:07.988035 11023 raft_consensus.cc:348] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } }
I20240501 14:09:07.988226 11023 raft_consensus.cc:729] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b42390c48f8c4e71a476d98115a685ac, State: Initialized, Role: LEARNER
I20240501 14:09:07.988575 11023 consensus_queue.cc:260] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } }
I20240501 14:09:07.988845 11036 tablet_bootstrap.cc:492] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Bootstrap starting.
I20240501 14:09:07.989388 11023 ts_tablet_manager.cc:1398] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
I20240501 14:09:07.990150 9202 tablet_copy_service.cc:342] P 90e73913f35c45dd8f2f517c3a385d3a: Request end of tablet copy session b42390c48f8c4e71a476d98115a685ac-ee739a5adf394e05b951d1e9c3ae5abe received from {username='slave'} at 127.6.82.5:38243
I20240501 14:09:07.990242 9202 tablet_copy_service.cc:433] P 90e73913f35c45dd8f2f517c3a385d3a: ending tablet copy session b42390c48f8c4e71a476d98115a685ac-ee739a5adf394e05b951d1e9c3ae5abe on tablet ee739a5adf394e05b951d1e9c3ae5abe with peer b42390c48f8c4e71a476d98115a685ac
I20240501 14:09:07.995975 11036 log.cc:824] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:07.999493 11015 tablet_bootstrap.cc:492] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:07.999769 11015 tablet_bootstrap.cc:492] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: Bootstrap complete.
I20240501 14:09:07.999912 11015 ts_tablet_manager.cc:1367] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20240501 14:09:08.000468 11015 raft_consensus.cc:348] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } }
I20240501 14:09:08.000622 11015 raft_consensus.cc:729] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06295523e9de4e2dae4b0909b0a04ea6, State: Initialized, Role: LEARNER
I20240501 14:09:08.000751 11015 consensus_queue.cc:260] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: true } }
I20240501 14:09:08.001034 11015 ts_tablet_manager.cc:1398] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:08.001818 9202 tablet_copy_service.cc:342] P 90e73913f35c45dd8f2f517c3a385d3a: Request end of tablet copy session 06295523e9de4e2dae4b0909b0a04ea6-06a2cf8244224085bccaf81a1c6cf85d received from {username='slave'} at 127.6.82.7:36269
I20240501 14:09:08.001911 9202 tablet_copy_service.cc:433] P 90e73913f35c45dd8f2f517c3a385d3a: ending tablet copy session 06295523e9de4e2dae4b0909b0a04ea6-06a2cf8244224085bccaf81a1c6cf85d on tablet 06a2cf8244224085bccaf81a1c6cf85d with peer 06295523e9de4e2dae4b0909b0a04ea6
I20240501 14:09:08.024168 11023 ts_tablet_manager.cc:902] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: Initiating tablet copy from peer b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:08.025084 11023 tablet_copy_client.cc:308] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:43063
I20240501 14:09:08.029713 11036 tablet_bootstrap.cc:492] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:08.030095 11036 tablet_bootstrap.cc:492] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Bootstrap complete.
I20240501 14:09:08.030123 9072 tablet_copy_service.cc:140] P b4c1a584bc6542f68b6d6912b091f1af: Received BeginTabletCopySession request for tablet d1d8e1ca84bc4b5980dc5a47db9412bf from peer b42390c48f8c4e71a476d98115a685ac ({username='slave'} at 127.6.82.5:34035)
I20240501 14:09:08.030220 9072 tablet_copy_service.cc:161] P b4c1a584bc6542f68b6d6912b091f1af: Beginning new tablet copy session on tablet d1d8e1ca84bc4b5980dc5a47db9412bf from peer b42390c48f8c4e71a476d98115a685ac at {username='slave'} at 127.6.82.5:34035: session id = b42390c48f8c4e71a476d98115a685ac-d1d8e1ca84bc4b5980dc5a47db9412bf
I20240501 14:09:08.030432 11036 ts_tablet_manager.cc:1367] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.001s
I20240501 14:09:08.031116 9072 tablet_copy_source_session.cc:205] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:08.032142 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d8e1ca84bc4b5980dc5a47db9412bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:08.033011 11036 raft_consensus.cc:348] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } }
I20240501 14:09:08.033301 11036 raft_consensus.cc:729] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa75a3f967ef4d259a6e56f2b457b39b, State: Initialized, Role: LEARNER
I20240501 14:09:08.033671 11036 consensus_queue.cc:260] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } }
I20240501 14:09:08.034636 11036 ts_tablet_manager.cc:1398] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240501 14:09:08.034852 11023 tablet_copy_client.cc:783] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:08.034993 11023 tablet_copy_client.cc:647] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:08.036440 9071 tablet_copy_service.cc:342] P b4c1a584bc6542f68b6d6912b091f1af: Request end of tablet copy session fa75a3f967ef4d259a6e56f2b457b39b-e138c8048df144d39157f2918473e43d received from {username='slave'} at 127.6.82.4:38267
I20240501 14:09:08.036513 9071 tablet_copy_service.cc:433] P b4c1a584bc6542f68b6d6912b091f1af: ending tablet copy session fa75a3f967ef4d259a6e56f2b457b39b-e138c8048df144d39157f2918473e43d on tablet e138c8048df144d39157f2918473e43d with peer fa75a3f967ef4d259a6e56f2b457b39b
I20240501 14:09:08.038384 11023 tablet_copy_client.cc:523] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:08.039693 11023 tablet_bootstrap.cc:492] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: Bootstrap starting.
I20240501 14:09:08.061049 11023 tablet_bootstrap.cc:492] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:08.061336 11023 tablet_bootstrap.cc:492] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: Bootstrap complete.
I20240501 14:09:08.061511 11023 ts_tablet_manager.cc:1367] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240501 14:09:08.062006 11023 raft_consensus.cc:348] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } }
I20240501 14:09:08.062111 11023 raft_consensus.cc:729] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b42390c48f8c4e71a476d98115a685ac, State: Initialized, Role: LEARNER
I20240501 14:09:08.062218 11023 consensus_queue.cc:260] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: true } }
I20240501 14:09:08.062494 11023 ts_tablet_manager.cc:1398] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:08.063148 9072 tablet_copy_service.cc:342] P b4c1a584bc6542f68b6d6912b091f1af: Request end of tablet copy session b42390c48f8c4e71a476d98115a685ac-d1d8e1ca84bc4b5980dc5a47db9412bf received from {username='slave'} at 127.6.82.5:34035
I20240501 14:09:08.063232 9072 tablet_copy_service.cc:433] P b4c1a584bc6542f68b6d6912b091f1af: ending tablet copy session b42390c48f8c4e71a476d98115a685ac-d1d8e1ca84bc4b5980dc5a47db9412bf on tablet d1d8e1ca84bc4b5980dc5a47db9412bf with peer b42390c48f8c4e71a476d98115a685ac
I20240501 14:09:08.098238 11036 ts_tablet_manager.cc:902] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: Initiating tablet copy from peer b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:08.098783 11036 tablet_copy_client.cc:308] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:43063
I20240501 14:09:08.099304 9072 tablet_copy_service.cc:140] P b4c1a584bc6542f68b6d6912b091f1af: Received BeginTabletCopySession request for tablet df4790f45f44444bb45df0877d5910a2 from peer fa75a3f967ef4d259a6e56f2b457b39b ({username='slave'} at 127.6.82.4:38267)
I20240501 14:09:08.099406 9072 tablet_copy_service.cc:161] P b4c1a584bc6542f68b6d6912b091f1af: Beginning new tablet copy session on tablet df4790f45f44444bb45df0877d5910a2 from peer fa75a3f967ef4d259a6e56f2b457b39b at {username='slave'} at 127.6.82.4:38267: session id = fa75a3f967ef4d259a6e56f2b457b39b-df4790f45f44444bb45df0877d5910a2
I20240501 14:09:08.100131 9072 tablet_copy_source_session.cc:205] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:08.100957 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4790f45f44444bb45df0877d5910a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:08.102597 11036 tablet_copy_client.cc:783] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:08.102763 11036 tablet_copy_client.cc:647] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:08.103760 11036 tablet_copy_client.cc:523] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:08.104665 11036 tablet_bootstrap.cc:492] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: Bootstrap starting.
I20240501 14:09:08.120999 11036 tablet_bootstrap.cc:492] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:08.121289 11036 tablet_bootstrap.cc:492] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: Bootstrap complete.
I20240501 14:09:08.121459 11036 ts_tablet_manager.cc:1367] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.001s
I20240501 14:09:08.121974 11036 raft_consensus.cc:348] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } }
I20240501 14:09:08.122079 11036 raft_consensus.cc:729] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa75a3f967ef4d259a6e56f2b457b39b, State: Initialized, Role: LEARNER
I20240501 14:09:08.122186 11036 consensus_queue.cc:260] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: true } }
I20240501 14:09:08.122407 11036 ts_tablet_manager.cc:1398] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:08.123056 9072 tablet_copy_service.cc:342] P b4c1a584bc6542f68b6d6912b091f1af: Request end of tablet copy session fa75a3f967ef4d259a6e56f2b457b39b-df4790f45f44444bb45df0877d5910a2 received from {username='slave'} at 127.6.82.4:38267
I20240501 14:09:08.123150 9072 tablet_copy_service.cc:433] P b4c1a584bc6542f68b6d6912b091f1af: ending tablet copy session fa75a3f967ef4d259a6e56f2b457b39b-df4790f45f44444bb45df0877d5910a2 on tablet df4790f45f44444bb45df0877d5910a2 with peer fa75a3f967ef4d259a6e56f2b457b39b
I20240501 14:09:08.292395 10583 raft_consensus.cc:1203] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.344820 10714 raft_consensus.cc:1203] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.362676 10171 raft_consensus.cc:1203] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.428385 10714 raft_consensus.cc:1203] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.454490 10321 raft_consensus.cc:1203] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.497098 10452 raft_consensus.cc:1203] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.535068 10452 raft_consensus.cc:1203] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.550645 10321 raft_consensus.cc:1203] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.559616 10583 raft_consensus.cc:1203] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:08.741356 11146 raft_consensus.cc:1050] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: attempting to promote NON_VOTER e32ae6129e5a47aa955c480ff1962e06 to VOTER
I20240501 14:09:08.741887 11146 consensus_queue.cc:237] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:08.743259 10583 raft_consensus.cc:1261] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Refusing update from remote peer b4c1a584bc6542f68b6d6912b091f1af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:08.743785 11154 consensus_queue.cc:1035] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.746524 11146 raft_consensus.cc:2932] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:08.747040 10583 raft_consensus.cc:2932] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:08.749100 8909 catalog_manager.cc:5484] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index 4 to 5, e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:08.766208 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a00189fb5d44f9b67639e8b8ed3303"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35496
I20240501 14:09:08.766409 9052 raft_consensus.cc:595] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:08.857592 11154 raft_consensus.cc:1050] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: attempting to promote NON_VOTER f2040b402c764007a4b1d969db917ede to VOTER
I20240501 14:09:08.857928 11154 consensus_queue.cc:237] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } }
I20240501 14:09:08.859181 10171 raft_consensus.cc:1261] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 LEARNER]: Refusing update from remote peer b4c1a584bc6542f68b6d6912b091f1af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:08.859618 11154 consensus_queue.cc:1035] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.861595 10863 raft_consensus.cc:2932] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f2040b402c764007a4b1d969db917ede (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } } }
I20240501 14:09:08.862097 10171 raft_consensus.cc:2932] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f2040b402c764007a4b1d969db917ede (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } } }
I20240501 14:09:08.864204 8903 catalog_manager.cc:5484] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index 4 to 5, f2040b402c764007a4b1d969db917ede (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:08.960642 11153 raft_consensus.cc:1050] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: attempting to promote NON_VOTER 06295523e9de4e2dae4b0909b0a04ea6 to VOTER
I20240501 14:09:08.961367 11153 consensus_queue.cc:237] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:08.962930 10714 raft_consensus.cc:1261] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Refusing update from remote peer 90e73913f35c45dd8f2f517c3a385d3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:08.963445 11152 consensus_queue.cc:1035] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.967074 10890 raft_consensus.cc:2932] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } } }
I20240501 14:09:08.967630 10714 raft_consensus.cc:2932] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } } }
I20240501 14:09:08.969488 8909 catalog_manager.cc:5484] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index 4 to 5, 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:08.979507 11153 raft_consensus.cc:1050] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: attempting to promote NON_VOTER e32ae6129e5a47aa955c480ff1962e06 to VOTER
I20240501 14:09:08.979841 11153 consensus_queue.cc:237] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:08.980945 10583 raft_consensus.cc:1261] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 LEARNER]: Refusing update from remote peer 90e73913f35c45dd8f2f517c3a385d3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:08.981335 11153 consensus_queue.cc:1035] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:08.982960 10890 raft_consensus.cc:2932] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:08.983377 10583 raft_consensus.cc:2932] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:08.985337 8909 catalog_manager.cc:5484] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 reported cstate change: config changed from index 4 to 5, e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:09.012697 10890 raft_consensus.cc:1050] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: attempting to promote NON_VOTER 06295523e9de4e2dae4b0909b0a04ea6 to VOTER
I20240501 14:09:09.013118 10890 consensus_queue.cc:237] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.014546 10714 raft_consensus.cc:1261] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 LEARNER]: Refusing update from remote peer 90e73913f35c45dd8f2f517c3a385d3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:09.015228 11153 consensus_queue.cc:1035] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:09.019731 10890 raft_consensus.cc:2932] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } } }
I20240501 14:09:09.020318 10714 raft_consensus.cc:2932] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } } }
I20240501 14:09:09.022367 8903 catalog_manager.cc:5484] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index 4 to 5, 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.028359 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a00189fb5d44f9b67639e8b8ed3303"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35566
I20240501 14:09:09.028506 9052 raft_consensus.cc:595] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.036772 11154 raft_consensus.cc:1050] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: attempting to promote NON_VOTER fa75a3f967ef4d259a6e56f2b457b39b to VOTER
I20240501 14:09:09.037202 11154 consensus_queue.cc:237] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.038547 10321 raft_consensus.cc:1261] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Refusing update from remote peer b4c1a584bc6542f68b6d6912b091f1af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:09.038990 11154 consensus_queue.cc:1035] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:09.040028 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a88e323853ac475a853775f585b71cc4"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34528
I20240501 14:09:09.040199 9183 raft_consensus.cc:595] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.042214 11146 raft_consensus.cc:2932] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } } }
I20240501 14:09:09.042899 10321 raft_consensus.cc:2932] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } } }
I20240501 14:09:09.045614 8909 catalog_manager.cc:5484] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index 4 to 5, fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.067121 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e138c8048df144d39157f2918473e43d"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35590
I20240501 14:09:09.067291 9052 raft_consensus.cc:595] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.077801 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7254f1aa0cd41e1bf8226d31c173095"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34562
I20240501 14:09:09.077950 9183 raft_consensus.cc:595] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.086568 10863 raft_consensus.cc:1050] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: attempting to promote NON_VOTER b42390c48f8c4e71a476d98115a685ac to VOTER
I20240501 14:09:09.086962 10863 consensus_queue.cc:237] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.088225 10452 raft_consensus.cc:1261] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Refusing update from remote peer b4c1a584bc6542f68b6d6912b091f1af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:09.088696 10863 consensus_queue.cc:1035] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:09.089061 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7316a1b44e054c96acc41b645b2d33c4"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35608
I20240501 14:09:09.089190 9052 raft_consensus.cc:595] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.091722 11154 raft_consensus.cc:2932] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b42390c48f8c4e71a476d98115a685ac (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:09.092260 10452 raft_consensus.cc:2932] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b42390c48f8c4e71a476d98115a685ac (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:09.094662 8903 catalog_manager.cc:5484] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index 4 to 5, b42390c48f8c4e71a476d98115a685ac (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.102752 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06a2cf8244224085bccaf81a1c6cf85d"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34582
I20240501 14:09:09.102905 9183 raft_consensus.cc:595] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.105448 11153 raft_consensus.cc:1050] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: attempting to promote NON_VOTER b42390c48f8c4e71a476d98115a685ac to VOTER
I20240501 14:09:09.105775 11153 consensus_queue.cc:237] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.107007 10452 raft_consensus.cc:1261] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 LEARNER]: Refusing update from remote peer 90e73913f35c45dd8f2f517c3a385d3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:09.107405 11152 consensus_queue.cc:1035] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:09.109118 11153 raft_consensus.cc:2932] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b42390c48f8c4e71a476d98115a685ac (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:09.109591 10452 raft_consensus.cc:2932] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b42390c48f8c4e71a476d98115a685ac (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:09.111743 8903 catalog_manager.cc:5484] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac reported cstate change: config changed from index 4 to 5, b42390c48f8c4e71a476d98115a685ac (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:09.166745 11154 raft_consensus.cc:1050] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: attempting to promote NON_VOTER fa75a3f967ef4d259a6e56f2b457b39b to VOTER
I20240501 14:09:09.167112 11154 consensus_queue.cc:237] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.168362 10321 raft_consensus.cc:1261] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 LEARNER]: Refusing update from remote peer b4c1a584bc6542f68b6d6912b091f1af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:09.168758 11154 consensus_queue.cc:1035] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:09.170760 10863 raft_consensus.cc:2932] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } } }
I20240501 14:09:09.171255 10321 raft_consensus.cc:2932] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } } }
I20240501 14:09:09.173408 8909 catalog_manager.cc:5484] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index 4 to 5, fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.288604 10863 raft_consensus.cc:979] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: : Instructing follower e32ae6129e5a47aa955c480ff1962e06 to start an election
I20240501 14:09:09.288734 10863 raft_consensus.cc:1067] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Signalling peer e32ae6129e5a47aa955c480ff1962e06 to start an election
I20240501 14:09:09.289363 10583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73a00189fb5d44f9b67639e8b8ed3303"
dest_uuid: "e32ae6129e5a47aa955c480ff1962e06"
from {username='slave'} at 127.6.82.1:36155
I20240501 14:09:09.289563 10583 raft_consensus.cc:482] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.289678 10583 raft_consensus.cc:3037] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.290589 10583 raft_consensus.cc:504] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.291551 10583 leader_election.cc:288] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [CANDIDATE]: Term 2 election: Requested vote from peers b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:09.292065 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a00189fb5d44f9b67639e8b8ed3303" candidate_uuid: "e32ae6129e5a47aa955c480ff1962e06" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
I20240501 14:09:09.292232 9052 raft_consensus.cc:3032] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.292280 9052 raft_consensus.cc:729] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:09.292457 9052 consensus_queue.cc:260] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.292616 9052 raft_consensus.cc:3037] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.293444 9052 raft_consensus.cc:2454] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32ae6129e5a47aa955c480ff1962e06 in term 2.
I20240501 14:09:09.293882 10516 leader_election.cc:302] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af, e32ae6129e5a47aa955c480ff1962e06; no voters:
I20240501 14:09:09.294124 11012 raft_consensus.cc:2781] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.294332 11012 raft_consensus.cc:686] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 2 LEADER]: Becoming Leader. State: Replica: e32ae6129e5a47aa955c480ff1962e06, State: Running, Role: LEADER
I20240501 14:09:09.294623 11012 consensus_queue.cc:237] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.296797 8903 catalog_manager.cc:5484] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 reported cstate change: term changed from 1 to 2, leader changed from b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) to e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6). New cstate: current_term: 2 leader_uuid: "e32ae6129e5a47aa955c480ff1962e06" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:09:09.303640 8897 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 73a00189fb5d44f9b67639e8b8ed3303 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6:34217) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:09:09.317842 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1d8e1ca84bc4b5980dc5a47db9412bf"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35630
I20240501 14:09:09.318004 9052 raft_consensus.cc:595] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.335963 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a88e323853ac475a853775f585b71cc4"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34594
I20240501 14:09:09.336124 9183 raft_consensus.cc:595] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.346967 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee739a5adf394e05b951d1e9c3ae5abe"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34616
I20240501 14:09:09.347115 9183 raft_consensus.cc:595] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.357805 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df4790f45f44444bb45df0877d5910a2"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35648
I20240501 14:09:09.357954 9052 raft_consensus.cc:595] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.369415 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e138c8048df144d39157f2918473e43d"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35660
I20240501 14:09:09.369587 9052 raft_consensus.cc:595] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.380291 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7254f1aa0cd41e1bf8226d31c173095"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34640
I20240501 14:09:09.380457 9183 raft_consensus.cc:595] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.391450 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7316a1b44e054c96acc41b645b2d33c4"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35664
I20240501 14:09:09.391628 9052 raft_consensus.cc:595] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.402765 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "06a2cf8244224085bccaf81a1c6cf85d"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34666
I20240501 14:09:09.402923 9183 raft_consensus.cc:595] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.444717 11153 raft_consensus.cc:979] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: : Instructing follower e32ae6129e5a47aa955c480ff1962e06 to start an election
I20240501 14:09:09.444842 11153 raft_consensus.cc:1067] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Signalling peer e32ae6129e5a47aa955c480ff1962e06 to start an election
I20240501 14:09:09.445493 10583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7254f1aa0cd41e1bf8226d31c173095"
dest_uuid: "e32ae6129e5a47aa955c480ff1962e06"
from {username='slave'} at 127.6.82.2:55457
I20240501 14:09:09.445648 10583 raft_consensus.cc:482] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.445719 10583 raft_consensus.cc:3037] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.446595 10583 raft_consensus.cc:504] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.447013 10583 leader_election.cc:288] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [CANDIDATE]: Term 2 election: Requested vote from peers 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:09.447474 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7254f1aa0cd41e1bf8226d31c173095" candidate_uuid: "e32ae6129e5a47aa955c480ff1962e06" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
I20240501 14:09:09.447626 9183 raft_consensus.cc:3032] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.447685 9183 raft_consensus.cc:729] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:09.447827 9183 consensus_queue.cc:260] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.447952 9183 raft_consensus.cc:3037] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.448705 9183 raft_consensus.cc:2454] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e32ae6129e5a47aa955c480ff1962e06 in term 2.
I20240501 14:09:09.449098 10517 leader_election.cc:302] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a, e32ae6129e5a47aa955c480ff1962e06; no voters:
I20240501 14:09:09.449343 11012 raft_consensus.cc:2781] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.449435 11012 raft_consensus.cc:686] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 2 LEADER]: Becoming Leader. State: Replica: e32ae6129e5a47aa955c480ff1962e06, State: Running, Role: LEADER
I20240501 14:09:09.449589 11012 consensus_queue.cc:237] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.451146 8903 catalog_manager.cc:5484] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 reported cstate change: term changed from 1 to 2, leader changed from 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) to e32ae6129e5a47aa955c480ff1962e06 (127.6.82.6). New cstate: current_term: 2 leader_uuid: "e32ae6129e5a47aa955c480ff1962e06" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.463671 11146 raft_consensus.cc:979] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: : Instructing follower f2040b402c764007a4b1d969db917ede to start an election
I20240501 14:09:09.463797 11146 raft_consensus.cc:1067] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Signalling peer f2040b402c764007a4b1d969db917ede to start an election
I20240501 14:09:09.464295 10171 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7316a1b44e054c96acc41b645b2d33c4"
dest_uuid: "f2040b402c764007a4b1d969db917ede"
from {username='slave'} at 127.6.82.1:55759
I20240501 14:09:09.464534 10171 raft_consensus.cc:482] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.464607 10171 raft_consensus.cc:3037] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.465504 10171 raft_consensus.cc:504] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } }
I20240501 14:09:09.466229 10171 leader_election.cc:288] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [CANDIDATE]: Term 2 election: Requested vote from peers b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:09.466652 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7316a1b44e054c96acc41b645b2d33c4" candidate_uuid: "f2040b402c764007a4b1d969db917ede" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
I20240501 14:09:09.466827 9052 raft_consensus.cc:3032] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.466882 9052 raft_consensus.cc:729] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:09.467067 9052 consensus_queue.cc:260] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } }
I20240501 14:09:09.467216 9052 raft_consensus.cc:3037] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.468055 9052 raft_consensus.cc:2454] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2040b402c764007a4b1d969db917ede in term 2.
I20240501 14:09:09.468442 10116 leader_election.cc:302] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af, f2040b402c764007a4b1d969db917ede; no voters:
I20240501 14:09:09.468775 11438 raft_consensus.cc:2781] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.468926 11438 raft_consensus.cc:686] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 2 LEADER]: Becoming Leader. State: Replica: f2040b402c764007a4b1d969db917ede, State: Running, Role: LEADER
I20240501 14:09:09.469067 11438 consensus_queue.cc:237] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } }
I20240501 14:09:09.470911 8903 catalog_manager.cc:5484] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede reported cstate change: term changed from 1 to 2, leader changed from b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) to f2040b402c764007a4b1d969db917ede (127.6.82.3). New cstate: current_term: 2 leader_uuid: "f2040b402c764007a4b1d969db917ede" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.510718 10890 raft_consensus.cc:979] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: : Instructing follower 06295523e9de4e2dae4b0909b0a04ea6 to start an election
I20240501 14:09:09.510906 10890 raft_consensus.cc:1067] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Signalling peer 06295523e9de4e2dae4b0909b0a04ea6 to start an election
I20240501 14:09:09.511461 10714 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a88e323853ac475a853775f585b71cc4"
dest_uuid: "06295523e9de4e2dae4b0909b0a04ea6"
from {username='slave'} at 127.6.82.2:38689
I20240501 14:09:09.511668 10714 raft_consensus.cc:482] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.511790 10714 raft_consensus.cc:3037] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.512681 10714 raft_consensus.cc:504] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.513597 10714 leader_election.cc:288] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [CANDIDATE]: Term 2 election: Requested vote from peers 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:09.514060 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88e323853ac475a853775f585b71cc4" candidate_uuid: "06295523e9de4e2dae4b0909b0a04ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
I20240501 14:09:09.514184 9183 raft_consensus.cc:3032] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.514237 9183 raft_consensus.cc:729] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:09.514407 9183 consensus_queue.cc:260] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.514552 9183 raft_consensus.cc:3037] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.515456 9183 raft_consensus.cc:2454] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06295523e9de4e2dae4b0909b0a04ea6 in term 2.
I20240501 14:09:09.515834 10648 leader_election.cc:302] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 06295523e9de4e2dae4b0909b0a04ea6, 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:09.516072 11261 raft_consensus.cc:2781] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.516266 11261 raft_consensus.cc:686] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 LEADER]: Becoming Leader. State: Replica: 06295523e9de4e2dae4b0909b0a04ea6, State: Running, Role: LEADER
I20240501 14:09:09.516566 11261 consensus_queue.cc:237] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.518944 8903 catalog_manager.cc:5484] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 reported cstate change: term changed from 1 to 2, leader changed from 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) to 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7). New cstate: current_term: 2 leader_uuid: "06295523e9de4e2dae4b0909b0a04ea6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.547282 11153 raft_consensus.cc:979] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: : Instructing follower 06295523e9de4e2dae4b0909b0a04ea6 to start an election
I20240501 14:09:09.547405 11153 raft_consensus.cc:1067] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Signalling peer 06295523e9de4e2dae4b0909b0a04ea6 to start an election
I20240501 14:09:09.547925 10714 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06a2cf8244224085bccaf81a1c6cf85d"
dest_uuid: "06295523e9de4e2dae4b0909b0a04ea6"
from {username='slave'} at 127.6.82.2:38689
I20240501 14:09:09.548061 10714 raft_consensus.cc:482] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.548127 10714 raft_consensus.cc:3037] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.548975 10714 raft_consensus.cc:504] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.549293 10714 leader_election.cc:288] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [CANDIDATE]: Term 2 election: Requested vote from peers 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:09.549966 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06a2cf8244224085bccaf81a1c6cf85d" candidate_uuid: "06295523e9de4e2dae4b0909b0a04ea6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
I20240501 14:09:09.550089 9183 raft_consensus.cc:3032] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.550140 9183 raft_consensus.cc:729] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:09.550318 9183 consensus_queue.cc:260] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.550451 9183 raft_consensus.cc:3037] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.551314 9183 raft_consensus.cc:2454] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06295523e9de4e2dae4b0909b0a04ea6 in term 2.
I20240501 14:09:09.551638 10648 leader_election.cc:302] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 06295523e9de4e2dae4b0909b0a04ea6, 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:09.551829 11261 raft_consensus.cc:2781] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.551924 11261 raft_consensus.cc:686] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 LEADER]: Becoming Leader. State: Replica: 06295523e9de4e2dae4b0909b0a04ea6, State: Running, Role: LEADER
I20240501 14:09:09.552065 11261 consensus_queue.cc:237] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:09.553529 8903 catalog_manager.cc:5484] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 reported cstate change: term changed from 1 to 2, leader changed from 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) to 06295523e9de4e2dae4b0909b0a04ea6 (127.6.82.7). New cstate: current_term: 2 leader_uuid: "06295523e9de4e2dae4b0909b0a04ea6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.614166 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1d8e1ca84bc4b5980dc5a47db9412bf"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59366
I20240501 14:09:09.614321 9052 raft_consensus.cc:595] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.625208 10863 raft_consensus.cc:979] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: : Instructing follower fa75a3f967ef4d259a6e56f2b457b39b to start an election
I20240501 14:09:09.625331 10863 raft_consensus.cc:1067] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Signalling peer fa75a3f967ef4d259a6e56f2b457b39b to start an election
I20240501 14:09:09.625869 10321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e138c8048df144d39157f2918473e43d"
dest_uuid: "fa75a3f967ef4d259a6e56f2b457b39b"
from {username='slave'} at 127.6.82.1:37451
I20240501 14:09:09.626084 10321 raft_consensus.cc:482] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.626230 10321 raft_consensus.cc:3037] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.627223 10321 raft_consensus.cc:504] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.628276 10321 leader_election.cc:288] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [CANDIDATE]: Term 2 election: Requested vote from peers b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:09.628793 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e138c8048df144d39157f2918473e43d" candidate_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
I20240501 14:09:09.628940 9052 raft_consensus.cc:3032] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.628996 9052 raft_consensus.cc:729] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:09.629158 9052 consensus_queue.cc:260] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.629305 9052 raft_consensus.cc:3037] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.630340 9052 raft_consensus.cc:2454] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa75a3f967ef4d259a6e56f2b457b39b in term 2.
I20240501 14:09:09.630743 10254 leader_election.cc:302] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af, fa75a3f967ef4d259a6e56f2b457b39b; no voters:
I20240501 14:09:09.631162 11084 raft_consensus.cc:2781] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.631350 11084 raft_consensus.cc:686] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 2 LEADER]: Becoming Leader. State: Replica: fa75a3f967ef4d259a6e56f2b457b39b, State: Running, Role: LEADER
I20240501 14:09:09.631639 11084 consensus_queue.cc:237] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.633713 8903 catalog_manager.cc:5484] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b reported cstate change: term changed from 1 to 2, leader changed from b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) to fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4). New cstate: current_term: 2 leader_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.641927 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee739a5adf394e05b951d1e9c3ae5abe"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43476
I20240501 14:09:09.642076 9183 raft_consensus.cc:595] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:09.648500 11146 raft_consensus.cc:979] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: : Instructing follower b42390c48f8c4e71a476d98115a685ac to start an election
I20240501 14:09:09.648628 11146 raft_consensus.cc:1067] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Signalling peer b42390c48f8c4e71a476d98115a685ac to start an election
I20240501 14:09:09.649120 10452 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1d8e1ca84bc4b5980dc5a47db9412bf"
dest_uuid: "b42390c48f8c4e71a476d98115a685ac"
from {username='slave'} at 127.6.82.1:38647
I20240501 14:09:09.649330 10452 raft_consensus.cc:482] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.649472 10452 raft_consensus.cc:3037] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.650408 10452 raft_consensus.cc:504] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.651345 10452 leader_election.cc:288] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [CANDIDATE]: Term 2 election: Requested vote from peers b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:09.651777 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d8e1ca84bc4b5980dc5a47db9412bf" candidate_uuid: "b42390c48f8c4e71a476d98115a685ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
I20240501 14:09:09.651922 9052 raft_consensus.cc:3032] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.651962 9052 raft_consensus.cc:729] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:09.652112 9052 consensus_queue.cc:260] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.652223 9052 raft_consensus.cc:3037] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.652623 10863 raft_consensus.cc:979] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: : Instructing follower fa75a3f967ef4d259a6e56f2b457b39b to start an election
I20240501 14:09:09.652720 10863 raft_consensus.cc:1067] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Signalling peer fa75a3f967ef4d259a6e56f2b457b39b to start an election
I20240501 14:09:09.653095 10321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df4790f45f44444bb45df0877d5910a2"
dest_uuid: "fa75a3f967ef4d259a6e56f2b457b39b"
from {username='slave'} at 127.6.82.1:37451
I20240501 14:09:09.653151 9052 raft_consensus.cc:2454] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42390c48f8c4e71a476d98115a685ac in term 2.
I20240501 14:09:09.653224 10321 raft_consensus.cc:482] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.653276 10321 raft_consensus.cc:3037] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.653551 10385 leader_election.cc:302] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b42390c48f8c4e71a476d98115a685ac, b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:09.653887 11485 raft_consensus.cc:2781] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.654079 11485 raft_consensus.cc:686] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 2 LEADER]: Becoming Leader. State: Replica: b42390c48f8c4e71a476d98115a685ac, State: Running, Role: LEADER
I20240501 14:09:09.654079 10321 raft_consensus.cc:504] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.654417 10321 leader_election.cc:288] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [CANDIDATE]: Term 2 election: Requested vote from peers b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:09.654387 11485 consensus_queue.cc:237] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.654801 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4790f45f44444bb45df0877d5910a2" candidate_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
I20240501 14:09:09.654919 9052 raft_consensus.cc:3032] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.654956 9052 raft_consensus.cc:729] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:09.655074 9052 consensus_queue.cc:260] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.655195 9052 raft_consensus.cc:3037] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.656255 9052 raft_consensus.cc:2454] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa75a3f967ef4d259a6e56f2b457b39b in term 2.
I20240501 14:09:09.656760 10254 leader_election.cc:302] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b4c1a584bc6542f68b6d6912b091f1af, fa75a3f967ef4d259a6e56f2b457b39b; no voters:
I20240501 14:09:09.656940 11084 raft_consensus.cc:2781] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.657022 11084 raft_consensus.cc:686] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 2 LEADER]: Becoming Leader. State: Replica: fa75a3f967ef4d259a6e56f2b457b39b, State: Running, Role: LEADER
I20240501 14:09:09.657147 11084 consensus_queue.cc:237] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:09.657343 8903 catalog_manager.cc:5484] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac reported cstate change: term changed from 1 to 2, leader changed from b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) to b42390c48f8c4e71a476d98115a685ac (127.6.82.5). New cstate: current_term: 2 leader_uuid: "b42390c48f8c4e71a476d98115a685ac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.657887 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df4790f45f44444bb45df0877d5910a2"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59378
I20240501 14:09:09.658020 9052 raft_consensus.cc:595] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 14:09:09.658071 9052 raft_consensus.cc:603] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 14:09:09.659390 8909 catalog_manager.cc:5484] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b reported cstate change: term changed from 1 to 2, leader changed from b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) to fa75a3f967ef4d259a6e56f2b457b39b (127.6.82.4). New cstate: current_term: 2 leader_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.680773 10890 raft_consensus.cc:979] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: : Instructing follower b42390c48f8c4e71a476d98115a685ac to start an election
I20240501 14:09:09.680914 10890 raft_consensus.cc:1067] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Signalling peer b42390c48f8c4e71a476d98115a685ac to start an election
I20240501 14:09:09.681406 10452 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee739a5adf394e05b951d1e9c3ae5abe"
dest_uuid: "b42390c48f8c4e71a476d98115a685ac"
from {username='slave'} at 127.6.82.2:56461
I20240501 14:09:09.681540 10452 raft_consensus.cc:482] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:09.681608 10452 raft_consensus.cc:3037] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.682468 10452 raft_consensus.cc:504] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.682960 10452 leader_election.cc:288] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [CANDIDATE]: Term 2 election: Requested vote from peers 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:09.683378 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee739a5adf394e05b951d1e9c3ae5abe" candidate_uuid: "b42390c48f8c4e71a476d98115a685ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
I20240501 14:09:09.683521 9183 raft_consensus.cc:3032] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:09.683568 9183 raft_consensus.cc:729] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:09.683728 9183 consensus_queue.cc:260] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.683861 9183 raft_consensus.cc:3037] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:09.684619 9183 raft_consensus.cc:2454] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42390c48f8c4e71a476d98115a685ac in term 2.
I20240501 14:09:09.684950 10386 leader_election.cc:302] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 90e73913f35c45dd8f2f517c3a385d3a, b42390c48f8c4e71a476d98115a685ac; no voters:
I20240501 14:09:09.685140 11485 raft_consensus.cc:2781] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:09.685276 11485 raft_consensus.cc:686] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 2 LEADER]: Becoming Leader. State: Replica: b42390c48f8c4e71a476d98115a685ac, State: Running, Role: LEADER
I20240501 14:09:09.685421 11485 consensus_queue.cc:237] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:09.686914 8909 catalog_manager.cc:5484] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac reported cstate change: term changed from 1 to 2, leader changed from 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) to b42390c48f8c4e71a476d98115a685ac (127.6.82.5). New cstate: current_term: 2 leader_uuid: "b42390c48f8c4e71a476d98115a685ac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.809175 9052 raft_consensus.cc:1261] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Refusing update from remote peer e32ae6129e5a47aa955c480ff1962e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:09.809907 11012 consensus_queue.cc:1035] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:09.936582 10583 consensus_queue.cc:237] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:09.937259 11012 raft_consensus.cc:2932] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:09.938802 8897 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73a00189fb5d44f9b67639e8b8ed3303 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:09.939952 8903 catalog_manager.cc:5484] T 73a00189fb5d44f9b67639e8b8ed3303 P e32ae6129e5a47aa955c480ff1962e06 reported cstate change: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "e32ae6129e5a47aa955c480ff1962e06" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:09.942857 9031 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a00189fb5d44f9b67639e8b8ed3303 with delete_type TABLET_DATA_TOMBSTONED (TS b4c1a584bc6542f68b6d6912b091f1af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35080
I20240501 14:09:09.943464 11551 tablet_replica.cc:330] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: stopping tablet replica
I20240501 14:09:09.943828 11551 raft_consensus.cc:2229] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:09.944016 11551 raft_consensus.cc:2258] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:09.945304 11551 ts_tablet_manager.cc:1875] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:09.947556 11551 ts_tablet_manager.cc:1888] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:09.947656 11551 log.cc:1196] T 73a00189fb5d44f9b67639e8b8ed3303 P b4c1a584bc6542f68b6d6912b091f1af: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/73a00189fb5d44f9b67639e8b8ed3303
I20240501 14:09:09.948313 8894 catalog_manager.cc:4830] TS b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063): tablet 73a00189fb5d44f9b67639e8b8ed3303 (table rebalance_test_table_4 [id=cbae0c7e67394a93876876fc67386fbb]) successfully deleted
I20240501 14:09:10.023119 9183 raft_consensus.cc:1261] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Refusing update from remote peer e32ae6129e5a47aa955c480ff1962e06: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.023762 11012 consensus_queue.cc:1035] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.037104 9052 raft_consensus.cc:1261] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Refusing update from remote peer fa75a3f967ef4d259a6e56f2b457b39b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.037714 11468 consensus_queue.cc:1035] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.051254 9183 raft_consensus.cc:1261] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Refusing update from remote peer 06295523e9de4e2dae4b0909b0a04ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.051806 11261 consensus_queue.cc:1035] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.059897 9183 raft_consensus.cc:1261] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Refusing update from remote peer 06295523e9de4e2dae4b0909b0a04ea6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.060405 11261 consensus_queue.cc:1035] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.075475 9052 raft_consensus.cc:1261] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Refusing update from remote peer f2040b402c764007a4b1d969db917ede: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.076098 11564 consensus_queue.cc:1035] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.086956 9052 raft_consensus.cc:1261] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Refusing update from remote peer fa75a3f967ef4d259a6e56f2b457b39b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.087445 11468 consensus_queue.cc:1035] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.092042 10583 consensus_queue.cc:237] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } }
I20240501 14:09:10.092656 11012 raft_consensus.cc:2932] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } } }
I20240501 14:09:10.093925 8897 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7254f1aa0cd41e1bf8226d31c173095 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.094815 8903 catalog_manager.cc:5484] T a7254f1aa0cd41e1bf8226d31c173095 P e32ae6129e5a47aa955c480ff1962e06 reported cstate change: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "e32ae6129e5a47aa955c480ff1962e06" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e32ae6129e5a47aa955c480ff1962e06" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 34217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.097381 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet a7254f1aa0cd41e1bf8226d31c173095 with delete_type TABLET_DATA_TOMBSTONED (TS 90e73913f35c45dd8f2f517c3a385d3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34110
I20240501 14:09:10.097921 11568 tablet_replica.cc:330] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: stopping tablet replica
I20240501 14:09:10.098038 10171 consensus_queue.cc:237] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } }
I20240501 14:09:10.098203 11568 raft_consensus.cc:2229] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.098390 11568 raft_consensus.cc:2258] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.098687 11564 raft_consensus.cc:2932] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } } }
I20240501 14:09:10.099951 8895 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7316a1b44e054c96acc41b645b2d33c4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.100787 11568 ts_tablet_manager.cc:1875] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.100849 8903 catalog_manager.cc:5484] T 7316a1b44e054c96acc41b645b2d33c4 P f2040b402c764007a4b1d969db917ede reported cstate change: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "f2040b402c764007a4b1d969db917ede" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f2040b402c764007a4b1d969db917ede" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 37685 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.103108 11568 ts_tablet_manager.cc:1888] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.103221 11568 log.cc:1196] T a7254f1aa0cd41e1bf8226d31c173095 P 90e73913f35c45dd8f2f517c3a385d3a: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/a7254f1aa0cd41e1bf8226d31c173095
I20240501 14:09:10.103371 9031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7316a1b44e054c96acc41b645b2d33c4 with delete_type TABLET_DATA_TOMBSTONED (TS b4c1a584bc6542f68b6d6912b091f1af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35080
I20240501 14:09:10.103639 11551 tablet_replica.cc:330] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: stopping tablet replica
I20240501 14:09:10.103798 11551 raft_consensus.cc:2229] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.103839 8895 catalog_manager.cc:4830] TS 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403): tablet a7254f1aa0cd41e1bf8226d31c173095 (table rebalance_test_table_1 [id=d838f67b9a3341e8a0d8ba0a99c0001c]) successfully deleted
I20240501 14:09:10.103929 11551 raft_consensus.cc:2258] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.104570 11551 ts_tablet_manager.cc:1875] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.106150 11551 ts_tablet_manager.cc:1888] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.106220 11551 log.cc:1196] T 7316a1b44e054c96acc41b645b2d33c4 P b4c1a584bc6542f68b6d6912b091f1af: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/7316a1b44e054c96acc41b645b2d33c4
I20240501 14:09:10.106704 8894 catalog_manager.cc:4830] TS b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063): tablet 7316a1b44e054c96acc41b645b2d33c4 (table rebalance_test_table_0 [id=ac384487a42c4c8aa5b9e8a68333b96f]) successfully deleted
I20240501 14:09:10.153323 9052 raft_consensus.cc:1261] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Refusing update from remote peer b42390c48f8c4e71a476d98115a685ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.153822 11485 consensus_queue.cc:1035] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.169077 10714 consensus_queue.cc:237] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:10.169749 11261 raft_consensus.cc:2932] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } } }
I20240501 14:09:10.171429 8897 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a88e323853ac475a853775f585b71cc4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.171908 8903 catalog_manager.cc:5484] T a88e323853ac475a853775f585b71cc4 P 06295523e9de4e2dae4b0909b0a04ea6 reported cstate change: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "06295523e9de4e2dae4b0909b0a04ea6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.174597 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet a88e323853ac475a853775f585b71cc4 with delete_type TABLET_DATA_TOMBSTONED (TS 90e73913f35c45dd8f2f517c3a385d3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34110
I20240501 14:09:10.174867 11568 tablet_replica.cc:330] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: stopping tablet replica
I20240501 14:09:10.174995 11568 raft_consensus.cc:2229] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.175079 11568 raft_consensus.cc:2258] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.175719 11568 ts_tablet_manager.cc:1875] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.177649 11568 ts_tablet_manager.cc:1888] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.177750 11568 log.cc:1196] T a88e323853ac475a853775f585b71cc4 P 90e73913f35c45dd8f2f517c3a385d3a: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/a88e323853ac475a853775f585b71cc4
I20240501 14:09:10.178345 8895 catalog_manager.cc:4830] TS 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403): tablet a88e323853ac475a853775f585b71cc4 (table rebalance_test_table_3 [id=3f15950b101c40918ad69275e08ab5af]) successfully deleted
I20240501 14:09:10.194494 10713 consensus_queue.cc:237] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } }
I20240501 14:09:10.195466 11561 raft_consensus.cc:2932] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } } }
I20240501 14:09:10.196676 8897 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 06a2cf8244224085bccaf81a1c6cf85d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.197626 8903 catalog_manager.cc:5484] T 06a2cf8244224085bccaf81a1c6cf85d P 06295523e9de4e2dae4b0909b0a04ea6 reported cstate change: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "06295523e9de4e2dae4b0909b0a04ea6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06295523e9de4e2dae4b0909b0a04ea6" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 42267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.198969 9183 raft_consensus.cc:1261] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Refusing update from remote peer b42390c48f8c4e71a476d98115a685ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:10.199486 11486 consensus_queue.cc:1035] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:10.201001 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet 06a2cf8244224085bccaf81a1c6cf85d with delete_type TABLET_DATA_TOMBSTONED (TS 90e73913f35c45dd8f2f517c3a385d3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34110
I20240501 14:09:10.201248 11568 tablet_replica.cc:330] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: stopping tablet replica
I20240501 14:09:10.201386 11568 raft_consensus.cc:2229] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.201494 11568 raft_consensus.cc:2258] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.202145 11568 ts_tablet_manager.cc:1875] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.204537 11568 ts_tablet_manager.cc:1888] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.204617 11568 log.cc:1196] T 06a2cf8244224085bccaf81a1c6cf85d P 90e73913f35c45dd8f2f517c3a385d3a: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/06a2cf8244224085bccaf81a1c6cf85d
I20240501 14:09:10.205089 8895 catalog_manager.cc:4830] TS 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403): tablet 06a2cf8244224085bccaf81a1c6cf85d (table rebalance_test_table_4 [id=cbae0c7e67394a93876876fc67386fbb]) successfully deleted
I20240501 14:09:10.267009 10321 consensus_queue.cc:237] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:10.267633 11084 raft_consensus.cc:2932] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } } }
I20240501 14:09:10.268932 8895 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df4790f45f44444bb45df0877d5910a2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.269785 8903 catalog_manager.cc:5484] T df4790f45f44444bb45df0877d5910a2 P fa75a3f967ef4d259a6e56f2b457b39b reported cstate change: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.272310 9031 tablet_service.cc:1514] Processing DeleteTablet for tablet df4790f45f44444bb45df0877d5910a2 with delete_type TABLET_DATA_TOMBSTONED (TS b4c1a584bc6542f68b6d6912b091f1af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35080
I20240501 14:09:10.272578 11551 tablet_replica.cc:330] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: stopping tablet replica
I20240501 14:09:10.272727 11551 raft_consensus.cc:2229] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.272840 11551 raft_consensus.cc:2258] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.273424 11551 ts_tablet_manager.cc:1875] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.275323 11551 ts_tablet_manager.cc:1888] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.275403 11551 log.cc:1196] T df4790f45f44444bb45df0877d5910a2 P b4c1a584bc6542f68b6d6912b091f1af: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/df4790f45f44444bb45df0877d5910a2
I20240501 14:09:10.275950 8894 catalog_manager.cc:4830] TS b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063): tablet df4790f45f44444bb45df0877d5910a2 (table rebalance_test_table_3 [id=3f15950b101c40918ad69275e08ab5af]) successfully deleted
I20240501 14:09:10.277130 10321 consensus_queue.cc:237] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } }
I20240501 14:09:10.277722 11084 raft_consensus.cc:2932] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } } }
I20240501 14:09:10.278944 8895 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e138c8048df144d39157f2918473e43d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.279834 8903 catalog_manager.cc:5484] T e138c8048df144d39157f2918473e43d P fa75a3f967ef4d259a6e56f2b457b39b reported cstate change: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fa75a3f967ef4d259a6e56f2b457b39b" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 33949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.282130 9031 tablet_service.cc:1514] Processing DeleteTablet for tablet e138c8048df144d39157f2918473e43d with delete_type TABLET_DATA_TOMBSTONED (TS b4c1a584bc6542f68b6d6912b091f1af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35080
I20240501 14:09:10.282387 11551 tablet_replica.cc:330] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: stopping tablet replica
I20240501 14:09:10.282532 11551 raft_consensus.cc:2229] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.282646 11551 raft_consensus.cc:2258] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.283290 11551 ts_tablet_manager.cc:1875] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.285370 11551 ts_tablet_manager.cc:1888] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.285470 11551 log.cc:1196] T e138c8048df144d39157f2918473e43d P b4c1a584bc6542f68b6d6912b091f1af: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/e138c8048df144d39157f2918473e43d
I20240501 14:09:10.286098 8894 catalog_manager.cc:4830] TS b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063): tablet e138c8048df144d39157f2918473e43d (table rebalance_test_table_0 [id=ac384487a42c4c8aa5b9e8a68333b96f]) successfully deleted
I20240501 14:09:10.296653 10452 consensus_queue.cc:237] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:10.297330 11485 raft_consensus.cc:2932] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:10.298650 8896 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1d8e1ca84bc4b5980dc5a47db9412bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.299607 8903 catalog_manager.cc:5484] T d1d8e1ca84bc4b5980dc5a47db9412bf P b42390c48f8c4e71a476d98115a685ac reported cstate change: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "b42390c48f8c4e71a476d98115a685ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.302160 9031 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d8e1ca84bc4b5980dc5a47db9412bf with delete_type TABLET_DATA_TOMBSTONED (TS b4c1a584bc6542f68b6d6912b091f1af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35080
I20240501 14:09:10.302390 11551 tablet_replica.cc:330] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: stopping tablet replica
I20240501 14:09:10.302510 11551 raft_consensus.cc:2229] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.302601 11551 raft_consensus.cc:2258] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.303263 11551 ts_tablet_manager.cc:1875] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.305104 11551 ts_tablet_manager.cc:1888] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.305176 11551 log.cc:1196] T d1d8e1ca84bc4b5980dc5a47db9412bf P b4c1a584bc6542f68b6d6912b091f1af: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/d1d8e1ca84bc4b5980dc5a47db9412bf
I20240501 14:09:10.305759 8894 catalog_manager.cc:4830] TS b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063): tablet d1d8e1ca84bc4b5980dc5a47db9412bf (table rebalance_test_table_2 [id=32df158a725c4ec1883752d3033e823c]) successfully deleted
I20240501 14:09:10.326373 10452 consensus_queue.cc:237] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } }
I20240501 14:09:10.327098 11485 raft_consensus.cc:2932] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } } }
I20240501 14:09:10.328492 8896 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee739a5adf394e05b951d1e9c3ae5abe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:10.329341 8903 catalog_manager.cc:5484] T ee739a5adf394e05b951d1e9c3ae5abe P b42390c48f8c4e71a476d98115a685ac reported cstate change: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "b42390c48f8c4e71a476d98115a685ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b42390c48f8c4e71a476d98115a685ac" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 46807 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:10.331980 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet ee739a5adf394e05b951d1e9c3ae5abe with delete_type TABLET_DATA_TOMBSTONED (TS 90e73913f35c45dd8f2f517c3a385d3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34110
I20240501 14:09:10.332257 11568 tablet_replica.cc:330] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: stopping tablet replica
I20240501 14:09:10.332394 11568 raft_consensus.cc:2229] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:10.332516 11568 raft_consensus.cc:2258] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:10.333124 11568 ts_tablet_manager.cc:1875] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:10.335002 11568 ts_tablet_manager.cc:1888] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:10.335076 11568 log.cc:1196] T ee739a5adf394e05b951d1e9c3ae5abe P 90e73913f35c45dd8f2f517c3a385d3a: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee739a5adf394e05b951d1e9c3ae5abe
I20240501 14:09:10.335613 8895 catalog_manager.cc:4830] TS 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403): tablet ee739a5adf394e05b951d1e9c3ae5abe (table rebalance_test_table_1 [id=d838f67b9a3341e8a0d8ba0a99c0001c]) successfully deleted
I20240501 14:09:10.572989 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.577838 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.587008 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.587527 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.588400 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.602027 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.622216 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.706697 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.707080 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.708827 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.714320 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.717177 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.723726 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.732728 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.990455 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.991758 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:10.993590 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:10.996354 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.001827 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.002686 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.028708 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.129793 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.133160 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.135744 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.138767 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.144910 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.159803 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.160197 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.280810 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.281747 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.285403 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.285951 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.291177 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.303840 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.305464 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
6308ae94f7da4aa6bb70f8e9cb63fceb | 127.6.82.62:37407 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.82.20:43523 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
06295523e9de4e2dae4b0909b0a04ea6 | 127.6.82.7:42267 | HEALTHY | <none> | 2 | 0
90e73913f35c45dd8f2f517c3a385d3a | 127.6.82.2:37403 | HEALTHY | <none> | 3 | 0
b42390c48f8c4e71a476d98115a685ac | 127.6.82.5:46807 | HEALTHY | <none> | 2 | 0
b4c1a584bc6542f68b6d6912b091f1af | 127.6.82.1:43063 | HEALTHY | <none> | 3 | 0
e32ae6129e5a47aa955c480ff1962e06 | 127.6.82.6:34217 | HEALTHY | <none> | 2 | 0
f2040b402c764007a4b1d969db917ede | 127.6.82.3:37685 | HEALTHY | <none> | 2 | 0
fa75a3f967ef4d259a6e56f2b457b39b | 127.6.82.4:33949 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.6.82.1 | experimental | 127.6.82.1:43063
local_ip_for_outbound_sockets | 127.6.82.2 | experimental | 127.6.82.2:37403
local_ip_for_outbound_sockets | 127.6.82.3 | experimental | 127.6.82.3:37685
local_ip_for_outbound_sockets | 127.6.82.4 | experimental | 127.6.82.4:33949
local_ip_for_outbound_sockets | 127.6.82.5 | experimental | 127.6.82.5:46807
local_ip_for_outbound_sockets | 127.6.82.6 | experimental | 127.6.82.6:34217
local_ip_for_outbound_sockets | 127.6.82.7 | experimental | 127.6.82.7:42267
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.82.1:43063
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.82.2:37403
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.82.3:37685
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.82.4:33949
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.82.5:46807
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.82.6:34217
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.82.7:42267
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.82.20:43523 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:11.350311 6472 log_verifier.cc:126] Checking tablet 04fc0a9f61d2446c868c2bc2a7792da7
I20240501 14:09:11.361809 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04fc0a9f61d2446c868c2bc2a7792da7
I20240501 14:09:11.361913 6472 log_verifier.cc:126] Checking tablet 06a2cf8244224085bccaf81a1c6cf85d
I20240501 14:09:11.370982 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 06a2cf8244224085bccaf81a1c6cf85d
I20240501 14:09:11.371066 6472 log_verifier.cc:126] Checking tablet 1202b0b9e74e4f4bbe7a31afe4fbf553
I20240501 14:09:11.379525 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1202b0b9e74e4f4bbe7a31afe4fbf553
I20240501 14:09:11.379606 6472 log_verifier.cc:126] Checking tablet 4472ad76c4fa46dc94e1824b2972a43d
I20240501 14:09:11.387866 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4472ad76c4fa46dc94e1824b2972a43d
I20240501 14:09:11.387943 6472 log_verifier.cc:126] Checking tablet 6a41ee08ad7141f5bb360dc992a3c072
I20240501 14:09:11.395356 6472 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6a41ee08ad7141f5bb360dc992a3c072
I20240501 14:09:11.395426 6472 log_verifier.cc:126] Checking tablet 7316a1b44e054c96acc41b645b2d33c4
I20240501 14:09:11.404183 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7316a1b44e054c96acc41b645b2d33c4
I20240501 14:09:11.404255 6472 log_verifier.cc:126] Checking tablet 73a00189fb5d44f9b67639e8b8ed3303
I20240501 14:09:11.413143 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 73a00189fb5d44f9b67639e8b8ed3303
I20240501 14:09:11.413220 6472 log_verifier.cc:126] Checking tablet 7a4aea6894f74532b1527bad663d9288
I20240501 14:09:11.421792 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7a4aea6894f74532b1527bad663d9288
I20240501 14:09:11.421869 6472 log_verifier.cc:126] Checking tablet 94ed7526967d4cafaa50839b73861a76
I20240501 14:09:11.430222 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 94ed7526967d4cafaa50839b73861a76
I20240501 14:09:11.430299 6472 log_verifier.cc:126] Checking tablet a7254f1aa0cd41e1bf8226d31c173095
I20240501 14:09:11.439236 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a7254f1aa0cd41e1bf8226d31c173095
I20240501 14:09:11.439311 6472 log_verifier.cc:126] Checking tablet a88e323853ac475a853775f585b71cc4
I20240501 14:09:11.448321 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a88e323853ac475a853775f585b71cc4
I20240501 14:09:11.448408 6472 log_verifier.cc:126] Checking tablet d1d8e1ca84bc4b5980dc5a47db9412bf
I20240501 14:09:11.457916 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d1d8e1ca84bc4b5980dc5a47db9412bf
I20240501 14:09:11.457995 6472 log_verifier.cc:126] Checking tablet df4790f45f44444bb45df0877d5910a2
I20240501 14:09:11.466894 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet df4790f45f44444bb45df0877d5910a2
I20240501 14:09:11.466967 6472 log_verifier.cc:126] Checking tablet dfb5160c34df401fb54fb75eab797b92
I20240501 14:09:11.475427 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dfb5160c34df401fb54fb75eab797b92
I20240501 14:09:11.475497 6472 log_verifier.cc:126] Checking tablet e138c8048df144d39157f2918473e43d
I20240501 14:09:11.484292 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e138c8048df144d39157f2918473e43d
I20240501 14:09:11.484370 6472 log_verifier.cc:126] Checking tablet ee739a5adf394e05b951d1e9c3ae5abe
I20240501 14:09:11.493018 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ee739a5adf394e05b951d1e9c3ae5abe
I20240501 14:09:11.495379 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.8:0
--local_ip_for_outbound_sockets=127.6.82.8
--webserver_interface=127.6.82.8
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--builtin_ntp_servers=127.6.82.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:11.611488 11855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:11.611763 11855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:11.611812 11855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:11.615206 11855 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:11.615303 11855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:11.615382 11855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.8
I20240501 14:09:11.619503 11855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.82.8
--webserver_port=0
--tserver_master_addrs=127.6.82.62:37407
--never_fsync=true
--heap_profile_path=/tmp/kudu.11855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.8
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:11.620558 11855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:11.621827 11855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:11.630534 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:11.630565 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:11.630534 11861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:11.630854 11855 server_base.cc:1034] running on GCE node
I20240501 14:09:11.631669 11855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:11.632396 11855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:11.633575 11855 hybrid_clock.cc:648] HybridClock initialized: now 1714572551633560 us; error 34 us; skew 500 ppm
I20240501 14:09:11.635880 11855 webserver.cc:460] Webserver started at http://127.6.82.8:44703/ using document root <none> and password file <none>
I20240501 14:09:11.636483 11855 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:11.636540 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:11.636704 11855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:11.638526 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5a0b99fc82c74822b621716d10e84cd3"
format_stamp: "Formatted at 2024-05-01 14:09:11 on dist-test-slave-crgf"
I20240501 14:09:11.639096 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5a0b99fc82c74822b621716d10e84cd3"
format_stamp: "Formatted at 2024-05-01 14:09:11 on dist-test-slave-crgf"
I20240501 14:09:11.642927 11855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:11.645534 11869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:11.646759 11855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:11.646903 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5a0b99fc82c74822b621716d10e84cd3"
format_stamp: "Formatted at 2024-05-01 14:09:11 on dist-test-slave-crgf"
I20240501 14:09:11.647003 11855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:11.664230 11855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:11.665035 11855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:11.665242 11855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:11.665858 11855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:11.666885 11855 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:11.666939 11855 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:11.667001 11855 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:11.667033 11855 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:11.676306 11855 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.8:42217
I20240501 14:09:11.676328 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.8:42217 every 8 connection(s)
I20240501 14:09:11.677343 11855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:11.682497 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 11855
I20240501 14:09:11.682610 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:09:11.687156 11983 heartbeater.cc:347] Connected to a master server at 127.6.82.62:37407
I20240501 14:09:11.687412 11983 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:11.688014 11983 heartbeater.cc:511] Master 127.6.82.62:37407 requested a full tablet report, sending...
I20240501 14:09:11.689221 8903 ts_manager.cc:194] Registered new tserver with Master: 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8:42217)
I20240501 14:09:11.690105 8903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.8:38539
I20240501 14:09:11.885629 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.889834 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.890794 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.890851 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:11.890836 11917 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:11.892673 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.902624 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:11.930353 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.049702 11917 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:12.051221 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.054579 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.064319 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.067411 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.069118 9163 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:12.077015 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:12.080960 9031 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:12.122313 9183 consensus_queue.cc:237] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } }
I20240501 14:09:12.123680 12076 raft_consensus.cc:2932] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } } }
W20240501 14:09:12.128732 9116 consensus_peers.cc:485] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a -> Peer 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8:42217): Couldn't send request to peer 5a0b99fc82c74822b621716d10e84cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94ed7526967d4cafaa50839b73861a76. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:12.131342 8910 catalog_manager.cc:5484] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a reported cstate change: config changed from index -1 to 4, NON_VOTER 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:12.134991 9052 consensus_queue.cc:237] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } }
I20240501 14:09:12.135948 12089 raft_consensus.cc:2932] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } } }
I20240501 14:09:12.139322 8910 catalog_manager.cc:5484] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index -1 to 4, NON_VOTER 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:12.142258 8985 consensus_peers.cc:485] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af -> Peer 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8:42217): Couldn't send request to peer 5a0b99fc82c74822b621716d10e84cd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04fc0a9f61d2446c868c2bc2a7792da7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:12.514854 12112 ts_tablet_manager.cc:902] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Initiating tablet copy from peer 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:12.515720 12112 tablet_copy_client.cc:308] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:37403
I20240501 14:09:12.519623 9202 tablet_copy_service.cc:140] P 90e73913f35c45dd8f2f517c3a385d3a: Received BeginTabletCopySession request for tablet 94ed7526967d4cafaa50839b73861a76 from peer 5a0b99fc82c74822b621716d10e84cd3 ({username='slave'} at 127.6.82.8:33309)
I20240501 14:09:12.519747 9202 tablet_copy_service.cc:161] P 90e73913f35c45dd8f2f517c3a385d3a: Beginning new tablet copy session on tablet 94ed7526967d4cafaa50839b73861a76 from peer 5a0b99fc82c74822b621716d10e84cd3 at {username='slave'} at 127.6.82.8:33309: session id = 5a0b99fc82c74822b621716d10e84cd3-94ed7526967d4cafaa50839b73861a76
I20240501 14:09:12.520545 9202 tablet_copy_source_session.cc:205] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:12.523769 12112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ed7526967d4cafaa50839b73861a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.529006 12112 tablet_copy_client.cc:783] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:12.529232 12112 tablet_copy_client.cc:647] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:12.531072 12112 tablet_copy_client.cc:523] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:12.532960 12112 tablet_bootstrap.cc:492] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Bootstrap starting.
I20240501 14:09:12.538626 12112 log.cc:824] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:12.560645 12112 tablet_bootstrap.cc:492] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:12.561080 12112 tablet_bootstrap.cc:492] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Bootstrap complete.
I20240501 14:09:12.561414 12112 ts_tablet_manager.cc:1367] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20240501 14:09:12.563514 12112 raft_consensus.cc:348] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } }
I20240501 14:09:12.563776 12112 raft_consensus.cc:729] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a0b99fc82c74822b621716d10e84cd3, State: Initialized, Role: LEARNER
I20240501 14:09:12.564122 12112 consensus_queue.cc:260] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } }
I20240501 14:09:12.564833 11983 heartbeater.cc:503] Master 127.6.82.62:37407 was elected leader, sending a full tablet report...
I20240501 14:09:12.565050 12112 ts_tablet_manager.cc:1398] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240501 14:09:12.566080 9202 tablet_copy_service.cc:342] P 90e73913f35c45dd8f2f517c3a385d3a: Request end of tablet copy session 5a0b99fc82c74822b621716d10e84cd3-94ed7526967d4cafaa50839b73861a76 received from {username='slave'} at 127.6.82.8:33309
I20240501 14:09:12.566188 9202 tablet_copy_service.cc:433] P 90e73913f35c45dd8f2f517c3a385d3a: ending tablet copy session 5a0b99fc82c74822b621716d10e84cd3-94ed7526967d4cafaa50839b73861a76 on tablet 94ed7526967d4cafaa50839b73861a76 with peer 5a0b99fc82c74822b621716d10e84cd3
I20240501 14:09:12.641330 12112 ts_tablet_manager.cc:902] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: Initiating tablet copy from peer b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:12.641860 12112 tablet_copy_client.cc:308] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:43063
I20240501 14:09:12.645095 9072 tablet_copy_service.cc:140] P b4c1a584bc6542f68b6d6912b091f1af: Received BeginTabletCopySession request for tablet 04fc0a9f61d2446c868c2bc2a7792da7 from peer 5a0b99fc82c74822b621716d10e84cd3 ({username='slave'} at 127.6.82.8:35885)
I20240501 14:09:12.645196 9072 tablet_copy_service.cc:161] P b4c1a584bc6542f68b6d6912b091f1af: Beginning new tablet copy session on tablet 04fc0a9f61d2446c868c2bc2a7792da7 from peer 5a0b99fc82c74822b621716d10e84cd3 at {username='slave'} at 127.6.82.8:35885: session id = 5a0b99fc82c74822b621716d10e84cd3-04fc0a9f61d2446c868c2bc2a7792da7
I20240501 14:09:12.645884 9072 tablet_copy_source_session.cc:205] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:12.646710 12112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04fc0a9f61d2446c868c2bc2a7792da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:12.648386 12112 tablet_copy_client.cc:783] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:12.648527 12112 tablet_copy_client.cc:647] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:12.649600 12112 tablet_copy_client.cc:523] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:12.650547 12112 tablet_bootstrap.cc:492] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: Bootstrap starting.
I20240501 14:09:12.667373 12112 tablet_bootstrap.cc:492] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:12.667639 12112 tablet_bootstrap.cc:492] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: Bootstrap complete.
I20240501 14:09:12.667810 12112 ts_tablet_manager.cc:1367] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.001s
I20240501 14:09:12.668318 12112 raft_consensus.cc:348] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } }
I20240501 14:09:12.668426 12112 raft_consensus.cc:729] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a0b99fc82c74822b621716d10e84cd3, State: Initialized, Role: LEARNER
I20240501 14:09:12.668522 12112 consensus_queue.cc:260] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: true } }
I20240501 14:09:12.668798 12112 ts_tablet_manager.cc:1398] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:12.669418 9072 tablet_copy_service.cc:342] P b4c1a584bc6542f68b6d6912b091f1af: Request end of tablet copy session 5a0b99fc82c74822b621716d10e84cd3-04fc0a9f61d2446c868c2bc2a7792da7 received from {username='slave'} at 127.6.82.8:35885
I20240501 14:09:12.669502 9072 tablet_copy_service.cc:433] P b4c1a584bc6542f68b6d6912b091f1af: ending tablet copy session 5a0b99fc82c74822b621716d10e84cd3-04fc0a9f61d2446c868c2bc2a7792da7 on tablet 04fc0a9f61d2446c868c2bc2a7792da7 with peer 5a0b99fc82c74822b621716d10e84cd3
I20240501 14:09:12.976258 11937 raft_consensus.cc:1203] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:13.046833 11937 raft_consensus.cc:1203] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:13.541697 12159 raft_consensus.cc:1050] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: attempting to promote NON_VOTER 5a0b99fc82c74822b621716d10e84cd3 to VOTER
I20240501 14:09:13.542102 12159 consensus_queue.cc:237] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:13.543562 11937 raft_consensus.cc:1261] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Refusing update from remote peer b4c1a584bc6542f68b6d6912b091f1af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:13.544015 12159 consensus_queue.cc:1035] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:13.546550 12130 raft_consensus.cc:2932] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:13.547108 11937 raft_consensus.cc:2932] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:13.549055 8903 catalog_manager.cc:5484] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af reported cstate change: config changed from index 4 to 5, 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4c1a584bc6542f68b6d6912b091f1af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:13.561272 12188 raft_consensus.cc:1050] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: attempting to promote NON_VOTER 5a0b99fc82c74822b621716d10e84cd3 to VOTER
I20240501 14:09:13.561640 12188 consensus_queue.cc:237] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:13.562692 11937 raft_consensus.cc:1261] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 LEARNER]: Refusing update from remote peer 90e73913f35c45dd8f2f517c3a385d3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:13.563088 12188 consensus_queue.cc:1035] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:13.564692 12187 raft_consensus.cc:2932] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:13.565063 11937 raft_consensus.cc:2932] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:13.567175 8903 catalog_manager.cc:5484] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 reported cstate change: config changed from index 4 to 5, 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90e73913f35c45dd8f2f517c3a385d3a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:13.639183 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04fc0a9f61d2446c868c2bc2a7792da7"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59526
I20240501 14:09:13.639333 9052 raft_consensus.cc:595] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:13.649438 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94ed7526967d4cafaa50839b73861a76"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43626
I20240501 14:09:13.649569 9183 raft_consensus.cc:595] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:13.860491 9052 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04fc0a9f61d2446c868c2bc2a7792da7"
dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59544
I20240501 14:09:13.860631 9052 raft_consensus.cc:595] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:13.869868 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94ed7526967d4cafaa50839b73861a76"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43642
I20240501 14:09:13.869992 9183 raft_consensus.cc:595] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:14.020139 12159 raft_consensus.cc:979] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: : Instructing follower 5a0b99fc82c74822b621716d10e84cd3 to start an election
I20240501 14:09:14.020244 12159 raft_consensus.cc:1067] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Signalling peer 5a0b99fc82c74822b621716d10e84cd3 to start an election
I20240501 14:09:14.020691 11937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04fc0a9f61d2446c868c2bc2a7792da7"
dest_uuid: "5a0b99fc82c74822b621716d10e84cd3"
from {username='slave'} at 127.6.82.1:48567
I20240501 14:09:14.020876 11937 raft_consensus.cc:482] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:14.020969 11937 raft_consensus.cc:3037] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:14.021997 11937 raft_consensus.cc:504] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.023036 11937 leader_election.cc:288] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [CANDIDATE]: Term 2 election: Requested vote from peers b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063)
I20240501 14:09:14.023427 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fc0a9f61d2446c868c2bc2a7792da7" candidate_uuid: "5a0b99fc82c74822b621716d10e84cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b4c1a584bc6542f68b6d6912b091f1af"
I20240501 14:09:14.023555 9052 raft_consensus.cc:3032] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:14.023602 9052 raft_consensus.cc:729] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4c1a584bc6542f68b6d6912b091f1af, State: Running, Role: LEADER
I20240501 14:09:14.023759 9052 consensus_queue.cc:260] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.023900 9052 raft_consensus.cc:3037] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:14.024618 9052 raft_consensus.cc:2454] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0b99fc82c74822b621716d10e84cd3 in term 2.
I20240501 14:09:14.024984 11870 leader_election.cc:302] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5a0b99fc82c74822b621716d10e84cd3, b4c1a584bc6542f68b6d6912b091f1af; no voters:
I20240501 14:09:14.025210 12182 raft_consensus.cc:2781] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:14.025408 12182 raft_consensus.cc:686] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 LEADER]: Becoming Leader. State: Replica: 5a0b99fc82c74822b621716d10e84cd3, State: Running, Role: LEADER
I20240501 14:09:14.025671 12182 consensus_queue.cc:237] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.027860 8910 catalog_manager.cc:5484] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 reported cstate change: term changed from 1 to 2, leader changed from b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) to 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8). New cstate: current_term: 2 leader_uuid: "5a0b99fc82c74822b621716d10e84cd3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:09:14.033856 8894 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 04fc0a9f61d2446c868c2bc2a7792da7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8:42217) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:09:14.085835 9183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94ed7526967d4cafaa50839b73861a76"
dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43664
I20240501 14:09:14.085965 9183 raft_consensus.cc:595] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:14.099581 12249 raft_consensus.cc:979] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: : Instructing follower 5a0b99fc82c74822b621716d10e84cd3 to start an election
I20240501 14:09:14.099678 12249 raft_consensus.cc:1067] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Signalling peer 5a0b99fc82c74822b621716d10e84cd3 to start an election
I20240501 14:09:14.100056 11937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94ed7526967d4cafaa50839b73861a76"
dest_uuid: "5a0b99fc82c74822b621716d10e84cd3"
from {username='slave'} at 127.6.82.2:34321
I20240501 14:09:14.100175 11937 raft_consensus.cc:482] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:14.100230 11937 raft_consensus.cc:3037] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:14.101020 11937 raft_consensus.cc:504] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.101393 11937 leader_election.cc:288] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [CANDIDATE]: Term 2 election: Requested vote from peers 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403)
I20240501 14:09:14.101727 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ed7526967d4cafaa50839b73861a76" candidate_uuid: "5a0b99fc82c74822b621716d10e84cd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90e73913f35c45dd8f2f517c3a385d3a"
I20240501 14:09:14.101857 9183 raft_consensus.cc:3032] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:14.101909 9183 raft_consensus.cc:729] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90e73913f35c45dd8f2f517c3a385d3a, State: Running, Role: LEADER
I20240501 14:09:14.102056 9183 consensus_queue.cc:260] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.102180 9183 raft_consensus.cc:3037] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:14.103081 9183 raft_consensus.cc:2454] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0b99fc82c74822b621716d10e84cd3 in term 2.
I20240501 14:09:14.103384 11871 leader_election.cc:302] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5a0b99fc82c74822b621716d10e84cd3, 90e73913f35c45dd8f2f517c3a385d3a; no voters:
I20240501 14:09:14.103543 12182 raft_consensus.cc:2781] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:14.103626 12182 raft_consensus.cc:686] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 LEADER]: Becoming Leader. State: Replica: 5a0b99fc82c74822b621716d10e84cd3, State: Running, Role: LEADER
I20240501 14:09:14.103780 12182 consensus_queue.cc:237] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.105165 8910 catalog_manager.cc:5484] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 reported cstate change: term changed from 1 to 2, leader changed from 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) to 5a0b99fc82c74822b621716d10e84cd3 (127.6.82.8). New cstate: current_term: 2 leader_uuid: "5a0b99fc82c74822b621716d10e84cd3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.505635 9183 raft_consensus.cc:1261] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Refusing update from remote peer 5a0b99fc82c74822b621716d10e84cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:14.506242 12182 consensus_queue.cc:1035] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e73913f35c45dd8f2f517c3a385d3a" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 37403 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:14.525154 9052 raft_consensus.cc:1261] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Refusing update from remote peer 5a0b99fc82c74822b621716d10e84cd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:14.525605 12231 consensus_queue.cc:1035] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c1a584bc6542f68b6d6912b091f1af" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 43063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:14.669867 11937 consensus_queue.cc:237] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.670506 12182 raft_consensus.cc:2932] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:14.671793 8894 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04fc0a9f61d2446c868c2bc2a7792da7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:14.672626 8910 catalog_manager.cc:5484] T 04fc0a9f61d2446c868c2bc2a7792da7 P 5a0b99fc82c74822b621716d10e84cd3 reported cstate change: config changed from index 5 to 7, VOTER b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "5a0b99fc82c74822b621716d10e84cd3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.675026 9031 tablet_service.cc:1514] Processing DeleteTablet for tablet 04fc0a9f61d2446c868c2bc2a7792da7 with delete_type TABLET_DATA_TOMBSTONED (TS b4c1a584bc6542f68b6d6912b091f1af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35080
I20240501 14:09:14.675364 12278 tablet_replica.cc:330] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: stopping tablet replica
I20240501 14:09:14.675482 12278 raft_consensus.cc:2229] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:14.675588 12278 raft_consensus.cc:2258] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:14.676883 12278 ts_tablet_manager.cc:1875] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:14.678633 12278 ts_tablet_manager.cc:1888] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:14.678699 12278 log.cc:1196] T 04fc0a9f61d2446c868c2bc2a7792da7 P b4c1a584bc6542f68b6d6912b091f1af: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/04fc0a9f61d2446c868c2bc2a7792da7
I20240501 14:09:14.679231 8894 catalog_manager.cc:4830] TS b4c1a584bc6542f68b6d6912b091f1af (127.6.82.1:43063): tablet 04fc0a9f61d2446c868c2bc2a7792da7 (table rebalance_test_table_1 [id=d838f67b9a3341e8a0d8ba0a99c0001c]) successfully deleted
I20240501 14:09:14.725847 11937 consensus_queue.cc:237] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } }
I20240501 14:09:14.726480 12182 raft_consensus.cc:2932] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } } }
I20240501 14:09:14.727706 8894 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94ed7526967d4cafaa50839b73861a76 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 14:09:14.728577 8910 catalog_manager.cc:5484] T 94ed7526967d4cafaa50839b73861a76 P 5a0b99fc82c74822b621716d10e84cd3 reported cstate change: config changed from index 5 to 7, VOTER 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "5a0b99fc82c74822b621716d10e84cd3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a0b99fc82c74822b621716d10e84cd3" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 42217 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:14.730774 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ed7526967d4cafaa50839b73861a76 with delete_type TABLET_DATA_TOMBSTONED (TS 90e73913f35c45dd8f2f517c3a385d3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34110
I20240501 14:09:14.731120 12294 tablet_replica.cc:330] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: stopping tablet replica
I20240501 14:09:14.731271 12294 raft_consensus.cc:2229] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:14.731382 12294 raft_consensus.cc:2258] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:14.732620 12294 ts_tablet_manager.cc:1875] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:14.734463 12294 ts_tablet_manager.cc:1888] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:14.734527 12294 log.cc:1196] T 94ed7526967d4cafaa50839b73861a76 P 90e73913f35c45dd8f2f517c3a385d3a: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/94ed7526967d4cafaa50839b73861a76
I20240501 14:09:14.735073 8895 catalog_manager.cc:4830] TS 90e73913f35c45dd8f2f517c3a385d3a (127.6.82.2:37403): tablet 94ed7526967d4cafaa50839b73861a76 (table rebalance_test_table_3 [id=3f15950b101c40918ad69275e08ab5af]) successfully deleted
I20240501 14:09:14.985826 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.986455 11917 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.995070 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.995541 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:14.996047 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.003858 9031 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.006346 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.010892 9163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.099849 9163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.100963 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.102167 11917 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.102648 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.105737 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.106678 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.111022 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.132856 9031 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.217289 10694 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.217507 11917 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.223985 9031 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.226836 10563 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.230064 9163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.233757 10432 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.240617 10301 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:15.246944 10153 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
6308ae94f7da4aa6bb70f8e9cb63fceb | 127.6.82.62:37407 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.82.20:43523 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
06295523e9de4e2dae4b0909b0a04ea6 | 127.6.82.7:42267 | HEALTHY | <none> | 2 | 0
5a0b99fc82c74822b621716d10e84cd3 | 127.6.82.8:42217 | HEALTHY | <none> | 2 | 0
90e73913f35c45dd8f2f517c3a385d3a | 127.6.82.2:37403 | HEALTHY | <none> | 2 | 0
b42390c48f8c4e71a476d98115a685ac | 127.6.82.5:46807 | HEALTHY | <none> | 2 | 0
b4c1a584bc6542f68b6d6912b091f1af | 127.6.82.1:43063 | HEALTHY | <none> | 2 | 0
e32ae6129e5a47aa955c480ff1962e06 | 127.6.82.6:34217 | HEALTHY | <none> | 2 | 0
f2040b402c764007a4b1d969db917ede | 127.6.82.3:37685 | HEALTHY | <none> | 2 | 0
fa75a3f967ef4d259a6e56f2b457b39b | 127.6.82.4:33949 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.6.82.1 | experimental | 127.6.82.1:43063
local_ip_for_outbound_sockets | 127.6.82.2 | experimental | 127.6.82.2:37403
local_ip_for_outbound_sockets | 127.6.82.3 | experimental | 127.6.82.3:37685
local_ip_for_outbound_sockets | 127.6.82.4 | experimental | 127.6.82.4:33949
local_ip_for_outbound_sockets | 127.6.82.5 | experimental | 127.6.82.5:46807
local_ip_for_outbound_sockets | 127.6.82.6 | experimental | 127.6.82.6:34217
local_ip_for_outbound_sockets | 127.6.82.7 | experimental | 127.6.82.7:42267
local_ip_for_outbound_sockets | 127.6.82.8 | experimental | 127.6.82.8:42217
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.82.1:43063
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.82.2:37403
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.82.3:37685
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.82.4:33949
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.82.5:46807
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.82.6:34217
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.82.7:42267
server_dump_info_path | /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.6.82.8:42217
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.82.20:43523 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:09:15.276453 6472 log_verifier.cc:126] Checking tablet 04fc0a9f61d2446c868c2bc2a7792da7
I20240501 14:09:15.285588 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 04fc0a9f61d2446c868c2bc2a7792da7
I20240501 14:09:15.285677 6472 log_verifier.cc:126] Checking tablet 06a2cf8244224085bccaf81a1c6cf85d
I20240501 14:09:15.291503 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 06a2cf8244224085bccaf81a1c6cf85d
I20240501 14:09:15.291564 6472 log_verifier.cc:126] Checking tablet 1202b0b9e74e4f4bbe7a31afe4fbf553
I20240501 14:09:15.297260 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1202b0b9e74e4f4bbe7a31afe4fbf553
I20240501 14:09:15.297318 6472 log_verifier.cc:126] Checking tablet 4472ad76c4fa46dc94e1824b2972a43d
I20240501 14:09:15.302945 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4472ad76c4fa46dc94e1824b2972a43d
I20240501 14:09:15.302997 6472 log_verifier.cc:126] Checking tablet 6a41ee08ad7141f5bb360dc992a3c072
I20240501 14:09:15.307641 6472 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6a41ee08ad7141f5bb360dc992a3c072
I20240501 14:09:15.307693 6472 log_verifier.cc:126] Checking tablet 7316a1b44e054c96acc41b645b2d33c4
I20240501 14:09:15.313958 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7316a1b44e054c96acc41b645b2d33c4
I20240501 14:09:15.314024 6472 log_verifier.cc:126] Checking tablet 73a00189fb5d44f9b67639e8b8ed3303
I20240501 14:09:15.320971 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 73a00189fb5d44f9b67639e8b8ed3303
I20240501 14:09:15.321030 6472 log_verifier.cc:126] Checking tablet 7a4aea6894f74532b1527bad663d9288
I20240501 14:09:15.327018 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7a4aea6894f74532b1527bad663d9288
I20240501 14:09:15.327061 6472 log_verifier.cc:126] Checking tablet 94ed7526967d4cafaa50839b73861a76
I20240501 14:09:15.335059 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 94ed7526967d4cafaa50839b73861a76
I20240501 14:09:15.335111 6472 log_verifier.cc:126] Checking tablet a7254f1aa0cd41e1bf8226d31c173095
I20240501 14:09:15.340498 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a7254f1aa0cd41e1bf8226d31c173095
I20240501 14:09:15.340548 6472 log_verifier.cc:126] Checking tablet a88e323853ac475a853775f585b71cc4
I20240501 14:09:15.346076 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a88e323853ac475a853775f585b71cc4
I20240501 14:09:15.346130 6472 log_verifier.cc:126] Checking tablet d1d8e1ca84bc4b5980dc5a47db9412bf
I20240501 14:09:15.351917 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d1d8e1ca84bc4b5980dc5a47db9412bf
I20240501 14:09:15.351964 6472 log_verifier.cc:126] Checking tablet df4790f45f44444bb45df0877d5910a2
I20240501 14:09:15.357533 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet df4790f45f44444bb45df0877d5910a2
I20240501 14:09:15.357580 6472 log_verifier.cc:126] Checking tablet dfb5160c34df401fb54fb75eab797b92
I20240501 14:09:15.362934 6472 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dfb5160c34df401fb54fb75eab797b92
I20240501 14:09:15.362980 6472 log_verifier.cc:126] Checking tablet e138c8048df144d39157f2918473e43d
I20240501 14:09:15.368470 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e138c8048df144d39157f2918473e43d
I20240501 14:09:15.368517 6472 log_verifier.cc:126] Checking tablet ee739a5adf394e05b951d1e9c3ae5abe
I20240501 14:09:15.373945 6472 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ee739a5adf394e05b951d1e9c3ae5abe
I20240501 14:09:15.374440 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8970
I20240501 14:09:15.381650 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 9101
I20240501 14:09:15.387990 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 10101
I20240501 14:09:15.393407 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 10232
I20240501 14:09:15.398732 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 10370
I20240501 14:09:15.404506 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 10501
I20240501 14:09:15.410260 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 10632
I20240501 14:09:15.415645 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 11855
I20240501 14:09:15.421216 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 8879
2024-05-01T14:09:15Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (15886 ms)
[----------] 1 test from RebalanceParamTest (15886 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240501 14:09:15.460108 6472 test_util.cc:268] Using random seed: 136060422
I20240501 14:09:15.460877 6472 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:09:15.460916 6472 ts_itest-base.cc:113] --------------
I20240501 14:09:15.460934 6472 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:09:15.460952 6472 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:09:15.460979 6472 ts_itest-base.cc:116] --------------
2024-05-01T14:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:09:15Z Disabled control of system clock
I20240501 14:09:15.469803 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:35905
--webserver_interface=127.6.82.62
--webserver_port=0
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.82.62:35905
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:09:15.572070 12414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.572274 12414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.572314 12414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.574987 12414 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.575037 12414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:09:15.575076 12414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.575093 12414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:09:15.575114 12414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:09:15.578490 12414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.82.62:35905
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.82.62:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.82.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.579274 12414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.580279 12414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.586973 12422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.586982 12420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.587121 12414 server_base.cc:1034] running on GCE node
W20240501 14:09:15.586992 12419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.587702 12414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.588539 12414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.589787 12414 hybrid_clock.cc:648] HybridClock initialized: now 1714572555589768 us; error 30 us; skew 500 ppm
I20240501 14:09:15.591675 12414 webserver.cc:460] Webserver started at http://127.6.82.62:42239/ using document root <none> and password file <none>
I20240501 14:09:15.592211 12414 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.592271 12414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.592468 12414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.594022 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bd843b84065446c09f71ac8534f21a8a"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.594470 12414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bd843b84065446c09f71ac8534f21a8a"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.597473 12414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:15.599572 12428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.600457 12414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.600567 12414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bd843b84065446c09f71ac8534f21a8a"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.600644 12414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.609954 12414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.610510 12414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.610659 12414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.616982 12414 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.62:35905
I20240501 14:09:15.616989 12480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.62:35905 every 8 connection(s)
I20240501 14:09:15.617878 12414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:09:15.620359 12481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:15.624591 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 12414
I20240501 14:09:15.624742 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:09:15.625272 12481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Bootstrap starting.
I20240501 14:09:15.627437 12481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:15.628159 12481 log.cc:824] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:15.629760 12481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: No bootstrap required, opened a new log
I20240501 14:09:15.632815 12481 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd843b84065446c09f71ac8534f21a8a" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 35905 } }
I20240501 14:09:15.633011 12481 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:15.633067 12481 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd843b84065446c09f71ac8534f21a8a, State: Initialized, Role: FOLLOWER
I20240501 14:09:15.633464 12481 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd843b84065446c09f71ac8534f21a8a" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 35905 } }
I20240501 14:09:15.633589 12481 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:09:15.633649 12481 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:09:15.633749 12481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:15.634411 12481 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd843b84065446c09f71ac8534f21a8a" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 35905 } }
I20240501 14:09:15.634701 12481 leader_election.cc:302] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd843b84065446c09f71ac8534f21a8a; no voters:
I20240501 14:09:15.634984 12481 leader_election.cc:288] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:09:15.635289 12486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:15.635478 12486 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [term 1 LEADER]: Becoming Leader. State: Replica: bd843b84065446c09f71ac8534f21a8a, State: Running, Role: LEADER
I20240501 14:09:15.635749 12486 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd843b84065446c09f71ac8534f21a8a" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 35905 } }
I20240501 14:09:15.635989 12481 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:09:15.637413 12488 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd843b84065446c09f71ac8534f21a8a. Latest consensus state: current_term: 1 leader_uuid: "bd843b84065446c09f71ac8534f21a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd843b84065446c09f71ac8534f21a8a" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 35905 } } }
I20240501 14:09:15.637446 12487 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd843b84065446c09f71ac8534f21a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd843b84065446c09f71ac8534f21a8a" member_type: VOTER last_known_addr { host: "127.6.82.62" port: 35905 } } }
I20240501 14:09:15.637516 12488 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:15.637533 12487 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a [sys.catalog]: This master's current role is: LEADER
I20240501 14:09:15.637845 12496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:09:15.640375 12496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:09:15.645272 12496 catalog_manager.cc:1347] Generated new cluster ID: 34250e935742488d82f391cb5ede91f2
I20240501 14:09:15.645337 12496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:09:15.668480 12496 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:09:15.669410 12496 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:09:15.675091 12496 catalog_manager.cc:5857] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Generated new TSK 0
I20240501 14:09:15.675776 12496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:09:15.686233 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.1:0
--local_ip_for_outbound_sockets=127.6.82.1
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:15.788571 12505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.788785 12505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.788828 12505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.791970 12505 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.792029 12505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.792124 12505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.1
I20240501 14:09:15.795610 12505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.82.1
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.12505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.1
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.796485 12505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.797555 12505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.805222 12513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.805259 12511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.805395 12505 server_base.cc:1034] running on GCE node
W20240501 14:09:15.805500 12510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.805876 12505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.806380 12505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.807541 12505 hybrid_clock.cc:648] HybridClock initialized: now 1714572555807516 us; error 38 us; skew 500 ppm
I20240501 14:09:15.809250 12505 webserver.cc:460] Webserver started at http://127.6.82.1:39935/ using document root <none> and password file <none>
I20240501 14:09:15.809727 12505 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.809777 12505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.809931 12505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.811534 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.811959 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.814998 12505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:15.817047 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.817978 12505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.818078 12505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.818150 12505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:15.839329 12505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:15.839951 12505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:15.840111 12505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:15.840638 12505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:15.841550 12505 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:15.841603 12505 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.841651 12505 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:15.841678 12505 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.849970 12505 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.1:46767
I20240501 14:09:15.849989 12632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.1:46767 every 8 connection(s)
I20240501 14:09:15.850966 12505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:09:15.851760 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 12505
I20240501 14:09:15.851889 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:09:15.854277 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.2:0
--local_ip_for_outbound_sockets=127.6.82.2
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:15.861215 12633 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:15.861451 12633 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:15.862015 12633 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:15.863658 12445 ts_manager.cc:194] Registered new tserver with Master: 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:15.865356 12445 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.1:60443
W20240501 14:09:15.960669 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:15.960865 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:15.960906 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:15.963825 12636 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:15.963897 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:15.963958 12636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.2
I20240501 14:09:15.967283 12636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.82.2
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.2
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:15.968063 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:15.969106 12636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:15.977119 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:15.977142 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.977226 12636 server_base.cc:1034] running on GCE node
W20240501 14:09:15.977146 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:15.977730 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:15.978257 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:15.979403 12636 hybrid_clock.cc:648] HybridClock initialized: now 1714572555979374 us; error 42 us; skew 500 ppm
I20240501 14:09:15.987246 12636 webserver.cc:460] Webserver started at http://127.6.82.2:36561/ using document root <none> and password file <none>
I20240501 14:09:15.987818 12636 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:15.987879 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:15.988045 12636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:15.989593 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.990049 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.993263 12636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:15.995375 12650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:15.996387 12636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:15.996492 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
format_stamp: "Formatted at 2024-05-01 14:09:15 on dist-test-slave-crgf"
I20240501 14:09:15.996578 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.014842 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.015450 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.015607 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.016150 12636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.017035 12636 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.017086 12636 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.017148 12636 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.017176 12636 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.025754 12636 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.2:40113
I20240501 14:09:16.025774 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.2:40113 every 8 connection(s)
I20240501 14:09:16.026690 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:09:16.031117 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 12636
I20240501 14:09:16.031273 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:09:16.033586 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.3:0
--local_ip_for_outbound_sockets=127.6.82.3
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:16.035794 12764 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:16.036008 12764 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.036509 12764 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:16.037650 12445 ts_manager.cc:194] Registered new tserver with Master: e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:16.038520 12445 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.2:33205
W20240501 14:09:16.139688 12767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.139914 12767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.139961 12767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.143148 12767 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:16.143222 12767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.143306 12767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.3
I20240501 14:09:16.146921 12767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.82.3
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.12767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.3
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.147783 12767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.148866 12767 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.156759 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.156713 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.156939 12767 server_base.cc:1034] running on GCE node
W20240501 14:09:16.157217 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.157552 12767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.158118 12767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.159281 12767 hybrid_clock.cc:648] HybridClock initialized: now 1714572556159270 us; error 23 us; skew 500 ppm
I20240501 14:09:16.161171 12767 webserver.cc:460] Webserver started at http://127.6.82.3:34235/ using document root <none> and password file <none>
I20240501 14:09:16.161705 12767 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.161758 12767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.161922 12767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.163553 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4902d2609e6d44d089590a42e70356b2"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.163966 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4902d2609e6d44d089590a42e70356b2"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.167071 12767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:16.169113 12781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.170074 12767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:16.170176 12767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4902d2609e6d44d089590a42e70356b2"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.170276 12767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.200083 12767 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.200690 12767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.200850 12767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.201397 12767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.202242 12767 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.202288 12767 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.202335 12767 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.202365 12767 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.210248 12767 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.3:42215
I20240501 14:09:16.210269 12894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.3:42215 every 8 connection(s)
I20240501 14:09:16.211195 12767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:09:16.219496 12895 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:16.219736 12895 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.219889 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 12767
I20240501 14:09:16.219976 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:09:16.220252 12895 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:16.221164 12444 ts_manager.cc:194] Registered new tserver with Master: 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:16.221807 12444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.3:44567
I20240501 14:09:16.221956 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.4:0
--local_ip_for_outbound_sockets=127.6.82.4
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:16.324419 12898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.324625 12898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.324700 12898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.327598 12898 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:16.327683 12898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.327770 12898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.4
I20240501 14:09:16.331138 12898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.82.4
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.12898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.4
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.331962 12898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.333019 12898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.340612 12903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.340624 12904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.340821 12898 server_base.cc:1034] running on GCE node
W20240501 14:09:16.340875 12906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.341288 12898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.341778 12898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.342923 12898 hybrid_clock.cc:648] HybridClock initialized: now 1714572556342908 us; error 29 us; skew 500 ppm
I20240501 14:09:16.344765 12898 webserver.cc:460] Webserver started at http://127.6.82.4:37085/ using document root <none> and password file <none>
I20240501 14:09:16.345285 12898 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.345343 12898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.345500 12898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.347079 12898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a6e94562bee149c2ace25672f0f9bf50"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.347513 12898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a6e94562bee149c2ace25672f0f9bf50"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.350524 12898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:16.352572 12912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.353453 12898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:16.353570 12898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a6e94562bee149c2ace25672f0f9bf50"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.353649 12898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.371981 12898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.372694 12898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.372869 12898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.373606 12898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.374737 12898 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.374820 12898 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.374884 12898 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.374917 12898 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.382594 12898 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.4:43151
I20240501 14:09:16.382670 13025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.4:43151 every 8 connection(s)
I20240501 14:09:16.383481 12898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:09:16.387319 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 12898
I20240501 14:09:16.387418 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:09:16.389603 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.5:0
--local_ip_for_outbound_sockets=127.6.82.5
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:16.392552 13026 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:16.392786 13026 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.393435 13026 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:16.394486 12444 ts_manager.cc:194] Registered new tserver with Master: a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:16.395023 12444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.4:54899
W20240501 14:09:16.497637 13029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.497853 13029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.497928 13029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.500902 13029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:16.500978 13029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.501061 13029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.5
I20240501 14:09:16.504478 13029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.82.5
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.13029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.5
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.505329 13029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.506348 13029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:16.514086 13034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.514197 13029 server_base.cc:1034] running on GCE node
W20240501 14:09:16.514240 13035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.514372 13037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.514782 13029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.515295 13029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.516443 13029 hybrid_clock.cc:648] HybridClock initialized: now 1714572556516431 us; error 23 us; skew 500 ppm
I20240501 14:09:16.518134 13029 webserver.cc:460] Webserver started at http://127.6.82.5:38729/ using document root <none> and password file <none>
I20240501 14:09:16.518684 13029 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.518771 13029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.518972 13029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.520992 13029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8a13c028ba8a4a659b74af6c265df95e"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.521581 13029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8a13c028ba8a4a659b74af6c265df95e"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.524899 13029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 14:09:16.527307 13043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.528658 13029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:16.528801 13029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8a13c028ba8a4a659b74af6c265df95e"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.528913 13029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.555413 13029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.555995 13029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.556130 13029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.556610 13029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.557439 13029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.557487 13029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.557524 13029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.557538 13029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.565528 13029 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.5:33177
I20240501 14:09:16.565569 13156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.5:33177 every 8 connection(s)
I20240501 14:09:16.566447 13029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:16.574941 13157 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:16.575145 13157 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.575664 13157 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:16.575676 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 13029
I20240501 14:09:16.575767 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:09:16.576611 12444 ts_manager.cc:194] Registered new tserver with Master: 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:16.577165 12444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.5:47131
I20240501 14:09:16.577630 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.6:0
--local_ip_for_outbound_sockets=127.6.82.6
--webserver_interface=127.6.82.6
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:16.683871 13160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.684082 13160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.684121 13160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.686851 13160 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:16.686911 13160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.686975 13160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.6
I20240501 14:09:16.690287 13160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.82.6
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.13160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.6
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.691190 13160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.692293 13160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:09:16.700165 13160 server_base.cc:1034] running on GCE node
W20240501 14:09:16.700002 13168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.700281 13165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.700006 13166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.700788 13160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.701395 13160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.702546 13160 hybrid_clock.cc:648] HybridClock initialized: now 1714572556702537 us; error 29 us; skew 500 ppm
I20240501 14:09:16.704429 13160 webserver.cc:460] Webserver started at http://127.6.82.6:33199/ using document root <none> and password file <none>
I20240501 14:09:16.704994 13160 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.705058 13160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.705237 13160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.706854 13160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "ab90e77f0027449bb3155980ca95593c"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.707326 13160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "ab90e77f0027449bb3155980ca95593c"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.710510 13160 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:16.712687 13174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.713622 13160 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:16.713725 13160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ab90e77f0027449bb3155980ca95593c"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.713820 13160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.727236 13160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.727772 13160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.727928 13160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.728427 13160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.729265 13160 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.729322 13160 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.729398 13160 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.729430 13160 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.737401 13160 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.6:37673
I20240501 14:09:16.737421 13287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.6:37673 every 8 connection(s)
I20240501 14:09:16.738328 13160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:16.743064 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 13160
I20240501 14:09:16.743184 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:09:16.745438 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.7:0
--local_ip_for_outbound_sockets=127.6.82.7
--webserver_interface=127.6.82.7
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:16.747077 13288 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:16.747310 13288 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.747819 13288 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:16.748744 12444 ts_manager.cc:194] Registered new tserver with Master: ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:16.749279 12444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.6:33225
W20240501 14:09:16.856074 13291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:16.856292 13291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:16.856350 13291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:16.859421 13291 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:16.859485 13291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:16.859551 13291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.7
I20240501 14:09:16.863186 13291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.82.7
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.13291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.7
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:16.864109 13291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:16.865144 13291 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 14:09:16.867713 12633 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
W20240501 14:09:16.872865 13296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:16.872879 13299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.873135 13291 server_base.cc:1034] running on GCE node
W20240501 14:09:16.873163 13297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:16.873534 13291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:16.874068 13291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:16.875219 13291 hybrid_clock.cc:648] HybridClock initialized: now 1714572556875203 us; error 33 us; skew 500 ppm
I20240501 14:09:16.877096 13291 webserver.cc:460] Webserver started at http://127.6.82.7:44773/ using document root <none> and password file <none>
I20240501 14:09:16.877625 13291 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:16.877698 13291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:16.877885 13291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:16.879547 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "66e2dc34330848c3b8c2dd15898d34c8"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.880019 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "66e2dc34330848c3b8c2dd15898d34c8"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.883278 13291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 14:09:16.885401 13305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.886410 13291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:16.886523 13291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "66e2dc34330848c3b8c2dd15898d34c8"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:16.886617 13291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:16.909227 13291 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:16.909811 13291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:16.909965 13291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:16.910465 13291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:16.911404 13291 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:16.911454 13291 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.911513 13291 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:16.911540 13291 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:16.919627 13291 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.7:34847
I20240501 14:09:16.919656 13418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.7:34847 every 8 connection(s)
I20240501 14:09:16.920558 13291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:16.921445 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 13291
I20240501 14:09:16.921566 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:09:16.924320 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.8:0
--local_ip_for_outbound_sockets=127.6.82.8
--webserver_interface=127.6.82.8
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:16.929472 13419 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:16.929739 13419 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:16.930478 13419 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:16.931682 12444 ts_manager.cc:194] Registered new tserver with Master: 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:16.932533 12444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.7:43845
W20240501 14:09:17.031976 13422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:17.032219 13422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:17.032267 13422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:17.035281 13422 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:17.035351 13422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:17.035415 13422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.8
I20240501 14:09:17.039147 13422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.82.8
--webserver_port=0
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.13422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.8
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:17.040030 13422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:17.040484 12764 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:17.041307 13422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:17.049080 13428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:17.049086 13430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:17.049347 13422 server_base.cc:1034] running on GCE node
W20240501 14:09:17.049371 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:17.049805 13422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:17.050349 13422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:17.051492 13422 hybrid_clock.cc:648] HybridClock initialized: now 1714572557051476 us; error 39 us; skew 500 ppm
I20240501 14:09:17.053372 13422 webserver.cc:460] Webserver started at http://127.6.82.8:41901/ using document root <none> and password file <none>
I20240501 14:09:17.053941 13422 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:17.054003 13422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:17.054172 13422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:09:17.055845 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "f23f426391724d6688aafffe7ee017d8"
format_stamp: "Formatted at 2024-05-01 14:09:17 on dist-test-slave-crgf"
I20240501 14:09:17.056313 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "f23f426391724d6688aafffe7ee017d8"
format_stamp: "Formatted at 2024-05-01 14:09:17 on dist-test-slave-crgf"
I20240501 14:09:17.059621 13422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:09:17.061759 13436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:17.062707 13422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:09:17.062845 13422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f23f426391724d6688aafffe7ee017d8"
format_stamp: "Formatted at 2024-05-01 14:09:17 on dist-test-slave-crgf"
I20240501 14:09:17.062927 13422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:17.079389 13422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:17.080018 13422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:17.080173 13422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:17.080673 13422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:17.081540 13422 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:17.081586 13422 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:17.081646 13422 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:17.081672 13422 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:17.089336 13422 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.8:46801
I20240501 14:09:17.089357 13549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.8:46801 every 8 connection(s)
I20240501 14:09:17.090296 13422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:17.098423 13550 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:17.098645 13550 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:17.099185 13550 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:17.100108 12444 ts_manager.cc:194] Registered new tserver with Master: f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:17.100138 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 13422
I20240501 14:09:17.100220 6472 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:09:17.100647 12444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.8:47219
I20240501 14:09:17.102595 6472 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:09:17.111749 6472 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:09:17.111824 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 13029
I20240501 14:09:17.117411 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 13160
I20240501 14:09:17.122328 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 13291
I20240501 14:09:17.127434 6472 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu with pid 13422
I20240501 14:09:17.223762 12895 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:17.397433 13026 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:20.884053 12444 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:20.904222 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 497040dcb4334bcd95fe5931f2b953f5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.904222 12959 tablet_service.cc:1467] Processing CreateTablet for tablet bebdcf50ed0046a29f4fbfac71087214 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.904222 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 547dce5435754bea84a0c68aa415e228 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.904222 12697 tablet_service.cc:1467] Processing CreateTablet for tablet bebdcf50ed0046a29f4fbfac71087214 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.905239 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497040dcb4334bcd95fe5931f2b953f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.905241 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebdcf50ed0046a29f4fbfac71087214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.905247 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebdcf50ed0046a29f4fbfac71087214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.905292 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 497040dcb4334bcd95fe5931f2b953f5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.905755 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 547dce5435754bea84a0c68aa415e228 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.905992 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 547dce5435754bea84a0c68aa415e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.906322 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497040dcb4334bcd95fe5931f2b953f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.909713 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 497040dcb4334bcd95fe5931f2b953f5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.910604 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497040dcb4334bcd95fe5931f2b953f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.911010 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 547dce5435754bea84a0c68aa415e228 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.911446 12565 tablet_service.cc:1467] Processing CreateTablet for tablet bebdcf50ed0046a29f4fbfac71087214 (DEFAULT_TABLE table=rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:20.911957 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 547dce5435754bea84a0c68aa415e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.912425 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 547dce5435754bea84a0c68aa415e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.913205 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebdcf50ed0046a29f4fbfac71087214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:20.914342 13572 tablet_bootstrap.cc:492] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:20.914698 13570 tablet_bootstrap.cc:492] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:20.916569 13572 tablet_bootstrap.cc:654] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.916687 13571 tablet_bootstrap.cc:492] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:20.917512 13572 log.cc:824] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.918120 13570 tablet_bootstrap.cc:654] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.918702 13571 tablet_bootstrap.cc:654] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.919031 13570 log.cc:824] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.919449 13571 log.cc:824] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.919759 13573 tablet_bootstrap.cc:492] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:20.921134 13570 tablet_bootstrap.cc:492] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:20.921149 13571 tablet_bootstrap.cc:492] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:20.921285 13572 tablet_bootstrap.cc:492] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:20.921312 13571 ts_tablet_manager.cc:1367] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:09:20.921377 13570 ts_tablet_manager.cc:1367] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:09:20.921521 13572 ts_tablet_manager.cc:1367] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:09:20.922585 13573 tablet_bootstrap.cc:654] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.923941 13573 log.cc:824] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:20.925444 13570 raft_consensus.cc:348] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:20.925578 13572 raft_consensus.cc:348] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.925767 13570 raft_consensus.cc:374] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.925855 13572 raft_consensus.cc:374] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.925879 13573 tablet_bootstrap.cc:492] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:20.925879 13570 raft_consensus.cc:729] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.925959 13572 raft_consensus.cc:729] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.926028 13573 ts_tablet_manager.cc:1367] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 14:09:20.926070 13571 raft_consensus.cc:348] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.926278 13571 raft_consensus.cc:374] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.926365 13571 raft_consensus.cc:729] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.926489 13572 consensus_queue.cc:260] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.926498 13570 consensus_queue.cc:260] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:20.926806 13571 consensus_queue.cc:260] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.927368 13572 ts_tablet_manager.cc:1398] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:09:20.927373 13570 ts_tablet_manager.cc:1398] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 14:09:20.927476 13571 ts_tablet_manager.cc:1398] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240501 14:09:20.927742 13572 tablet_bootstrap.cc:492] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:20.928846 13572 tablet_bootstrap.cc:654] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.929684 13571 tablet_bootstrap.cc:492] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:20.930262 13572 tablet_bootstrap.cc:492] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:20.930354 13572 ts_tablet_manager.cc:1367] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:20.930936 13572 raft_consensus.cc:348] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:20.930882 13573 raft_consensus.cc:348] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.931030 13572 raft_consensus.cc:374] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.931066 13573 raft_consensus.cc:374] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.931070 13572 raft_consensus.cc:729] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.931129 13573 raft_consensus.cc:729] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.931206 13572 consensus_queue.cc:260] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:20.931501 13572 ts_tablet_manager.cc:1398] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.931540 13573 consensus_queue.cc:260] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.932422 13573 ts_tablet_manager.cc:1398] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240501 14:09:20.932736 13571 tablet_bootstrap.cc:654] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.932991 13573 tablet_bootstrap.cc:492] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:20.933950 13573 tablet_bootstrap.cc:654] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.934300 13571 tablet_bootstrap.cc:492] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:20.934376 13571 ts_tablet_manager.cc:1367] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:09:20.934686 13570 tablet_bootstrap.cc:492] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:20.934979 13571 raft_consensus.cc:348] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.935077 13571 raft_consensus.cc:374] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.935110 13571 raft_consensus.cc:729] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.935230 13573 tablet_bootstrap.cc:492] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:20.935223 13571 consensus_queue.cc:260] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.935308 13573 ts_tablet_manager.cc:1367] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:20.935499 13571 ts_tablet_manager.cc:1398] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.935694 13570 tablet_bootstrap.cc:654] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.935943 13573 raft_consensus.cc:348] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:20.936094 13573 raft_consensus.cc:374] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.936146 13573 raft_consensus.cc:729] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.936312 13573 consensus_queue.cc:260] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:20.936702 13573 ts_tablet_manager.cc:1398] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:20.936898 13573 tablet_bootstrap.cc:492] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:20.936898 13570 tablet_bootstrap.cc:492] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:20.936983 13570 ts_tablet_manager.cc:1367] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:20.937528 13570 raft_consensus.cc:348] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.937629 13570 raft_consensus.cc:374] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.937662 13570 raft_consensus.cc:729] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.937762 13573 tablet_bootstrap.cc:654] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:20.937778 13570 consensus_queue.cc:260] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.938067 13570 ts_tablet_manager.cc:1398] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.939119 13573 tablet_bootstrap.cc:492] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:20.939255 13573 ts_tablet_manager.cc:1367] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:20.939646 13573 raft_consensus.cc:348] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.939792 13573 raft_consensus.cc:374] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:20.939839 13573 raft_consensus.cc:729] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:20.939960 13573 consensus_queue.cc:260] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.940192 13573 ts_tablet_manager.cc:1398] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:20.946549 13580 raft_consensus.cc:482] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:20.946707 13580 raft_consensus.cc:504] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.947722 13580 leader_election.cc:288] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:20.952239 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547dce5435754bea84a0c68aa415e228" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:20.952240 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547dce5435754bea84a0c68aa415e228" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:20.952508 12718 raft_consensus.cc:2454] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:20.952508 12587 raft_consensus.cc:2454] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:20.952944 12782 leader_election.cc:302] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:20.953169 13580 raft_consensus.cc:2781] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:20.953238 13580 raft_consensus.cc:482] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:20.953266 13580 raft_consensus.cc:3037] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.954088 13580 raft_consensus.cc:504] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.954401 13580 leader_election.cc:288] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:20.954695 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547dce5435754bea84a0c68aa415e228" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:20.954699 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "547dce5435754bea84a0c68aa415e228" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:20.954829 12718 raft_consensus.cc:3037] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.954824 12587 raft_consensus.cc:3037] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:20.955698 12718 raft_consensus.cc:2454] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:20.955974 12782 leader_election.cc:302] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:20.956043 12587 raft_consensus.cc:2454] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:20.956334 13580 raft_consensus.cc:2781] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:20.956542 13580 raft_consensus.cc:686] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:20.956857 13580 consensus_queue.cc:237] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:20.959614 12444 catalog_manager.cc:5484] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:20.965267 12896 tablet.cc:2369] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:21.031369 12765 tablet.cc:2369] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:21.082015 13582 raft_consensus.cc:482] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.082165 13582 raft_consensus.cc:504] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:21.083360 13582 leader_election.cc:288] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:21.087985 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdcf50ed0046a29f4fbfac71087214" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:21.088146 12718 raft_consensus.cc:2454] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:21.088150 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdcf50ed0046a29f4fbfac71087214" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:21.088465 12980 raft_consensus.cc:2454] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:21.088500 12520 leader_election.cc:302] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:21.088798 13582 raft_consensus.cc:2781] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.088872 13582 raft_consensus.cc:482] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.088917 13582 raft_consensus.cc:3037] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.089720 13582 raft_consensus.cc:504] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:21.090052 13582 leader_election.cc:288] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:21.090343 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdcf50ed0046a29f4fbfac71087214" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:21.090370 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebdcf50ed0046a29f4fbfac71087214" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:21.090459 12980 raft_consensus.cc:3037] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.090497 12718 raft_consensus.cc:3037] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.091382 12718 raft_consensus.cc:2454] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:21.091403 12980 raft_consensus.cc:2454] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:21.091665 12520 leader_election.cc:302] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:21.091861 13582 raft_consensus.cc:2781] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.092078 13582 raft_consensus.cc:686] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:21.092378 13582 consensus_queue.cc:237] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:21.095289 12444 catalog_manager.cc:5484] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:21.106089 12634 tablet.cc:2369] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:21.137706 13027 tablet.cc:2369] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:21.141109 13580 raft_consensus.cc:482] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.141218 13580 raft_consensus.cc:504] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.141641 13580 leader_election.cc:288] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:21.142403 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497040dcb4334bcd95fe5931f2b953f5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:21.142557 12587 raft_consensus.cc:2454] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:21.142887 12784 leader_election.cc:302] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:21.143067 13580 raft_consensus.cc:2781] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.143118 13580 raft_consensus.cc:482] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.143149 13580 raft_consensus.cc:3037] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.144511 13580 raft_consensus.cc:504] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.144893 13580 leader_election.cc:288] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:21.145236 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497040dcb4334bcd95fe5931f2b953f5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:21.145354 12587 raft_consensus.cc:3037] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.145428 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497040dcb4334bcd95fe5931f2b953f5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:21.145480 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497040dcb4334bcd95fe5931f2b953f5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:21.145558 12980 raft_consensus.cc:2454] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:21.146286 12587 raft_consensus.cc:2454] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:21.146554 12784 leader_election.cc:302] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters: a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:21.146720 13580 raft_consensus.cc:2781] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.146860 13580 raft_consensus.cc:686] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:21.147003 13580 consensus_queue.cc:237] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.148547 12444 catalog_manager.cc:5484] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:21.307370 13596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:21.307612 13596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:21.310499 13596 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:09:21.401495 13580 consensus_queue.cc:1035] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.409583 13589 consensus_queue.cc:1035] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.420492 12587 raft_consensus.cc:1261] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.421615 12979 raft_consensus.cc:3037] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.422864 12979 raft_consensus.cc:1261] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.423821 13589 consensus_queue.cc:1035] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.424649 13580 consensus_queue.cc:1035] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.431991 12980 raft_consensus.cc:1261] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.432643 13582 consensus_queue.cc:1035] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.433822 12717 raft_consensus.cc:1261] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:21.434964 13582 consensus_queue.cc:1035] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:21.443416 13624 mvcc.cc:205] Tried to move back new op lower bound from 7022889211571507200 to 7022889210459160576. Current Snapshot: MvccSnapshot[applied={T|T < 7022889211571507200}]
I20240501 14:09:21.448864 13625 mvcc.cc:205] Tried to move back new op lower bound from 7022889211613466624 to 7022889210236747776. Current Snapshot: MvccSnapshot[applied={T|T < 7022889211613466624}]
I20240501 14:09:21.451606 13629 mvcc.cc:205] Tried to move back new op lower bound from 7022889211571507200 to 7022889210459160576. Current Snapshot: MvccSnapshot[applied={T|T < 7022889211571507200}]
INSERT report
rows total: 2000
time total: 153.557 ms
time per row: 0.0767783 ms
I20240501 14:09:21.520121 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:21.526355 12698 tablet_service.cc:1467] Processing CreateTablet for tablet a5957da39df64253a2b8d7f388308719 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.526748 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5957da39df64253a2b8d7f388308719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.527163 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 66436017fd044f0b919629933ea56e9c (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.527343 12829 tablet_service.cc:1467] Processing CreateTablet for tablet a5957da39df64253a2b8d7f388308719 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.527469 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66436017fd044f0b919629933ea56e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.527592 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5957da39df64253a2b8d7f388308719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.529848 12959 tablet_service.cc:1467] Processing CreateTablet for tablet a5957da39df64253a2b8d7f388308719 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.530006 13649 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:21.530133 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5957da39df64253a2b8d7f388308719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.531038 12960 tablet_service.cc:1467] Processing CreateTablet for tablet d64234effb3446bba1ff82bdf37acaa7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.531247 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64234effb3446bba1ff82bdf37acaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.531562 13649 tablet_bootstrap.cc:654] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.532896 13649 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:21.532987 13649 ts_tablet_manager.cc:1367] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:21.533576 13649 raft_consensus.cc:348] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.533726 12698 tablet_service.cc:1467] Processing CreateTablet for tablet d64234effb3446bba1ff82bdf37acaa7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.534286 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 66436017fd044f0b919629933ea56e9c (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.534533 12829 tablet_service.cc:1467] Processing CreateTablet for tablet d64234effb3446bba1ff82bdf37acaa7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.534067 12697 tablet_service.cc:1467] Processing CreateTablet for tablet 66436017fd044f0b919629933ea56e9c (DEFAULT_TABLE table=rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:21.534253 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64234effb3446bba1ff82bdf37acaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.534587 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66436017fd044f0b919629933ea56e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.535135 13650 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:21.536119 13650 tablet_bootstrap.cc:654] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.537514 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66436017fd044f0b919629933ea56e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.537590 13651 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:21.537670 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64234effb3446bba1ff82bdf37acaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:21.538584 13651 tablet_bootstrap.cc:654] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.533892 13649 raft_consensus.cc:374] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.540009 13649 raft_consensus.cc:729] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.540218 13649 consensus_queue.cc:260] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.540612 13650 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:21.540642 13649 ts_tablet_manager.cc:1398] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:21.540709 13650 ts_tablet_manager.cc:1367] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:21.540867 13649 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:21.541167 13650 raft_consensus.cc:348] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.541263 13650 raft_consensus.cc:374] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.541304 13650 raft_consensus.cc:729] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.541774 13650 consensus_queue.cc:260] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.542130 13650 ts_tablet_manager.cc:1398] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:21.542187 13649 tablet_bootstrap.cc:654] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.542395 13650 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:21.543072 13652 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:21.543879 13650 tablet_bootstrap.cc:654] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.544262 13652 tablet_bootstrap.cc:654] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.545524 13649 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:21.545607 13649 ts_tablet_manager.cc:1367] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:21.546005 13651 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:21.546098 13651 ts_tablet_manager.cc:1367] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:21.546124 13649 raft_consensus.cc:348] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.546274 13649 raft_consensus.cc:374] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.546324 13649 raft_consensus.cc:729] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.546466 13649 consensus_queue.cc:260] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.546540 13651 raft_consensus.cc:348] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.546653 13651 raft_consensus.cc:374] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.546694 13651 raft_consensus.cc:729] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.546842 13651 consensus_queue.cc:260] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.547086 13651 ts_tablet_manager.cc:1398] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:21.547662 13649 ts_tablet_manager.cc:1398] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:21.547884 13649 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:21.548826 13649 tablet_bootstrap.cc:654] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.550099 13650 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:21.550199 13650 ts_tablet_manager.cc:1367] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:21.550570 13650 raft_consensus.cc:348] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.550662 13650 raft_consensus.cc:374] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.550694 13650 raft_consensus.cc:729] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.550858 13650 consensus_queue.cc:260] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.551096 13650 ts_tablet_manager.cc:1398] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:21.551251 13650 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:21.552183 13650 tablet_bootstrap.cc:654] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.553422 13650 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:21.553501 13650 ts_tablet_manager.cc:1367] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:21.553917 13650 raft_consensus.cc:348] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.554014 13650 raft_consensus.cc:374] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.554061 13650 raft_consensus.cc:729] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.554194 13650 consensus_queue.cc:260] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.554459 13650 ts_tablet_manager.cc:1398] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:21.555449 13649 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:21.555495 13652 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:21.555541 13649 ts_tablet_manager.cc:1367] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:21.555593 13652 ts_tablet_manager.cc:1367] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:21.555986 13649 raft_consensus.cc:348] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.556087 13649 raft_consensus.cc:374] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.556118 13649 raft_consensus.cc:729] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.556149 13652 raft_consensus.cc:348] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.556303 13652 raft_consensus.cc:374] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.556339 13652 raft_consensus.cc:729] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.556347 13649 consensus_queue.cc:260] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.556520 13652 consensus_queue.cc:260] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.556701 13649 ts_tablet_manager.cc:1398] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:21.556859 13652 ts_tablet_manager.cc:1398] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:21.557044 13652 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:21.558245 13652 tablet_bootstrap.cc:654] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:21.560523 13652 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:21.560611 13652 ts_tablet_manager.cc:1367] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:21.561170 13652 raft_consensus.cc:348] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.561318 13652 raft_consensus.cc:374] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:21.561368 13652 raft_consensus.cc:729] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:21.561506 13652 consensus_queue.cc:260] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.561869 13652 ts_tablet_manager.cc:1398] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:21.708124 13647 raft_consensus.cc:482] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.708283 13647 raft_consensus.cc:504] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.708781 13647 leader_election.cc:288] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:21.709249 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:21.709264 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:21.709398 12717 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:21.709406 12980 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:21.709736 12784 leader_election.cc:302] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:21.709915 13647 raft_consensus.cc:2781] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.709975 13647 raft_consensus.cc:482] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.710005 13647 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.710762 13647 raft_consensus.cc:504] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.711035 13647 leader_election.cc:288] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:21.711386 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:21.711410 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:21.711493 12717 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.711517 12980 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.712385 12980 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:21.712605 12717 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:21.712713 12784 leader_election.cc:302] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:21.712899 13647 raft_consensus.cc:2781] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.712996 13647 raft_consensus.cc:686] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:21.713152 13647 consensus_queue.cc:237] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:21.714763 12443 catalog_manager.cc:5484] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:21.790589 13631 raft_consensus.cc:482] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.790750 13631 raft_consensus.cc:504] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.791254 13631 leader_election.cc:288] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:21.791800 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66436017fd044f0b919629933ea56e9c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:21.791965 12717 raft_consensus.cc:2454] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:21.792289 12520 leader_election.cc:302] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:21.792464 13631 raft_consensus.cc:2781] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.792529 13631 raft_consensus.cc:482] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.792557 13631 raft_consensus.cc:3037] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.793354 13631 raft_consensus.cc:504] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.793673 13631 leader_election.cc:288] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:21.794134 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66436017fd044f0b919629933ea56e9c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:21.794261 12717 raft_consensus.cc:3037] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.795164 12717 raft_consensus.cc:2454] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:21.795432 12520 leader_election.cc:302] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:21.795600 13631 raft_consensus.cc:2781] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.795681 13631 raft_consensus.cc:686] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:21.795526 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66436017fd044f0b919629933ea56e9c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:21.795822 13631 consensus_queue.cc:237] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:21.795874 12849 raft_consensus.cc:2454] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:21.795527 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66436017fd044f0b919629933ea56e9c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:21.796175 12848 raft_consensus.cc:3037] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.797127 12848 raft_consensus.cc:2454] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:21.797350 12443 catalog_manager.cc:5484] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:21.868507 13647 raft_consensus.cc:482] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:21.868629 13647 raft_consensus.cc:504] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.869024 13647 leader_election.cc:288] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:21.869324 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64234effb3446bba1ff82bdf37acaa7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:21.869367 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64234effb3446bba1ff82bdf37acaa7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:21.869472 12717 raft_consensus.cc:2454] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:21.869545 12980 raft_consensus.cc:2454] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:21.869774 12782 leader_election.cc:302] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:21.869938 13647 raft_consensus.cc:2781] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:21.869999 13647 raft_consensus.cc:482] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:21.870047 13647 raft_consensus.cc:3037] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.870862 13647 raft_consensus.cc:504] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.871156 13647 leader_election.cc:288] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:21.871464 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64234effb3446bba1ff82bdf37acaa7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:21.871464 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64234effb3446bba1ff82bdf37acaa7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:21.871575 12980 raft_consensus.cc:3037] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.871575 12717 raft_consensus.cc:3037] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:21.872488 12717 raft_consensus.cc:2454] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:21.872488 12980 raft_consensus.cc:2454] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:21.872786 12784 leader_election.cc:302] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:21.872954 13647 raft_consensus.cc:2781] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:21.873055 13647 raft_consensus.cc:686] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:21.873180 13647 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:21.874831 12443 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:22.009160 13666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:22.009409 13666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:22.012519 13666 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:09:22.107230 12848 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.107760 12717 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.108336 13582 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.108471 13631 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.110114 12980 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.111246 13589 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.113777 12718 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.114182 13647 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.114995 12979 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.117671 13647 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.118769 12716 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.120028 13647 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.123533 13699 mvcc.cc:205] Tried to move back new op lower bound from 7022889214385606656 to 7022889213116768256. Current Snapshot: MvccSnapshot[applied={T|T < 7022889214385606656}]
INSERT report
rows total: 2000
time total: 125.495 ms
time per row: 0.0627477 ms
I20240501 14:09:22.182842 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:22.188504 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 5b30c755056741dda7ab43f07aaf2609 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.188548 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 5b30c755056741dda7ab43f07aaf2609 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.188594 12697 tablet_service.cc:1467] Processing CreateTablet for tablet 5b30c755056741dda7ab43f07aaf2609 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.188828 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b30c755056741dda7ab43f07aaf2609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.188850 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b30c755056741dda7ab43f07aaf2609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.188848 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b30c755056741dda7ab43f07aaf2609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.189242 12698 tablet_service.cc:1467] Processing CreateTablet for tablet eca34c8d867f4f2293ad5aac5fb8aadb (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.189452 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca34c8d867f4f2293ad5aac5fb8aadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.191550 12959 tablet_service.cc:1467] Processing CreateTablet for tablet eca34c8d867f4f2293ad5aac5fb8aadb (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.191629 12565 tablet_service.cc:1467] Processing CreateTablet for tablet eca34c8d867f4f2293ad5aac5fb8aadb (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.191794 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca34c8d867f4f2293ad5aac5fb8aadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.191880 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca34c8d867f4f2293ad5aac5fb8aadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.191627 12567 tablet_service.cc:1467] Processing CreateTablet for tablet cbeba1e7c473403da87b5a0cd3c70d7e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.192024 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbeba1e7c473403da87b5a0cd3c70d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.192322 12829 tablet_service.cc:1467] Processing CreateTablet for tablet cbeba1e7c473403da87b5a0cd3c70d7e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.192576 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbeba1e7c473403da87b5a0cd3c70d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.194051 12960 tablet_service.cc:1467] Processing CreateTablet for tablet cbeba1e7c473403da87b5a0cd3c70d7e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.194200 13729 tablet_bootstrap.cc:492] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:22.194275 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbeba1e7c473403da87b5a0cd3c70d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.194433 13728 tablet_bootstrap.cc:492] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:22.195075 13729 tablet_bootstrap.cc:654] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.195360 13731 tablet_bootstrap.cc:492] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:22.195401 13728 tablet_bootstrap.cc:654] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.196192 13729 tablet_bootstrap.cc:492] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:22.196226 13731 tablet_bootstrap.cc:654] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.196281 13729 ts_tablet_manager.cc:1367] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:22.191809 13730 tablet_bootstrap.cc:492] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:22.196743 13729 raft_consensus.cc:348] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.196978 13729 raft_consensus.cc:374] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.197027 13729 raft_consensus.cc:729] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.197216 13729 consensus_queue.cc:260] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.197299 13731 tablet_bootstrap.cc:492] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:22.197376 13731 ts_tablet_manager.cc:1367] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:22.197543 13730 tablet_bootstrap.cc:654] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.197556 13729 ts_tablet_manager.cc:1398] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.197597 13728 tablet_bootstrap.cc:492] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:22.197679 13728 ts_tablet_manager.cc:1367] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.197778 13729 tablet_bootstrap.cc:492] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:22.197854 13731 raft_consensus.cc:348] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.197950 13731 raft_consensus.cc:374] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.197984 13731 raft_consensus.cc:729] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.198150 13731 consensus_queue.cc:260] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.198150 13728 raft_consensus.cc:348] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.198279 13728 raft_consensus.cc:374] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.198549 13731 ts_tablet_manager.cc:1398] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.198606 13730 tablet_bootstrap.cc:492] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:22.198679 13730 ts_tablet_manager.cc:1367] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:22.198758 13729 tablet_bootstrap.cc:654] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.199105 13730 raft_consensus.cc:348] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.199201 13730 raft_consensus.cc:374] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.199250 13730 raft_consensus.cc:729] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.199359 13730 consensus_queue.cc:260] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.199632 13730 ts_tablet_manager.cc:1398] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.199798 13730 tablet_bootstrap.cc:492] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:22.198318 13728 raft_consensus.cc:729] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.200619 13728 consensus_queue.cc:260] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.201745 13728 ts_tablet_manager.cc:1398] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:22.202001 13728 tablet_bootstrap.cc:492] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:22.203056 13728 tablet_bootstrap.cc:654] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.203161 13730 tablet_bootstrap.cc:654] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.204542 13729 tablet_bootstrap.cc:492] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:22.204636 13729 ts_tablet_manager.cc:1367] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:22.204775 13730 tablet_bootstrap.cc:492] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:22.204849 13730 ts_tablet_manager.cc:1367] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:22.204943 13728 tablet_bootstrap.cc:492] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:22.205011 13728 ts_tablet_manager.cc:1367] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.205102 13729 raft_consensus.cc:348] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.205196 13729 raft_consensus.cc:374] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.205235 13729 raft_consensus.cc:729] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.205302 13730 raft_consensus.cc:348] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.205348 13729 consensus_queue.cc:260] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.205399 13730 raft_consensus.cc:374] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.205430 13730 raft_consensus.cc:729] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.205580 13729 ts_tablet_manager.cc:1398] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.205554 13730 consensus_queue.cc:260] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.205721 13729 tablet_bootstrap.cc:492] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:22.205827 13730 ts_tablet_manager.cc:1398] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:22.206010 13730 tablet_bootstrap.cc:492] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:22.206001 13728 raft_consensus.cc:348] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.206094 13728 raft_consensus.cc:374] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.206122 13728 raft_consensus.cc:729] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.206247 13728 consensus_queue.cc:260] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.206569 13729 tablet_bootstrap.cc:654] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.206578 13728 ts_tablet_manager.cc:1398] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:22.206858 13730 tablet_bootstrap.cc:654] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.207862 13730 tablet_bootstrap.cc:492] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:22.207974 13730 ts_tablet_manager.cc:1367] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:22.208417 13730 raft_consensus.cc:348] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.208520 13730 raft_consensus.cc:374] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.208554 13730 raft_consensus.cc:729] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.208628 13729 tablet_bootstrap.cc:492] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:22.208731 13729 ts_tablet_manager.cc:1367] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:22.208731 13730 consensus_queue.cc:260] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.209030 13730 ts_tablet_manager.cc:1398] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:22.209179 13729 raft_consensus.cc:348] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.209359 13729 raft_consensus.cc:374] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.209440 13729 raft_consensus.cc:729] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.209610 13729 consensus_queue.cc:260] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.209892 13729 ts_tablet_manager.cc:1398] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:22.270054 13695 raft_consensus.cc:482] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.270197 13695 raft_consensus.cc:504] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.271256 13695 leader_election.cc:288] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.275561 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b30c755056741dda7ab43f07aaf2609" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:22.275734 12587 raft_consensus.cc:2454] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:22.276082 12653 leader_election.cc:302] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:22.276255 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b30c755056741dda7ab43f07aaf2609" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:22.276341 13695 raft_consensus.cc:2781] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.276413 13695 raft_consensus.cc:482] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.276403 12980 raft_consensus.cc:2454] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:22.276465 13695 raft_consensus.cc:3037] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.277345 13695 raft_consensus.cc:504] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.277666 13695 leader_election.cc:288] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.277976 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b30c755056741dda7ab43f07aaf2609" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:22.278048 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b30c755056741dda7ab43f07aaf2609" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:22.278093 12587 raft_consensus.cc:3037] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.278185 12980 raft_consensus.cc:3037] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.278898 12587 raft_consensus.cc:2454] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:22.278996 12980 raft_consensus.cc:2454] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:22.279167 12653 leader_election.cc:302] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:22.279325 13695 raft_consensus.cc:2781] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.279521 13695 raft_consensus.cc:686] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:22.279803 13695 consensus_queue.cc:237] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.281888 12443 catalog_manager.cc:5484] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:22.306874 13709 raft_consensus.cc:482] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.307024 13709 raft_consensus.cc:504] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.308228 13709 leader_election.cc:288] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:22.311750 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca34c8d867f4f2293ad5aac5fb8aadb" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:22.311772 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca34c8d867f4f2293ad5aac5fb8aadb" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:22.311893 12587 raft_consensus.cc:2454] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:22.311923 12716 raft_consensus.cc:2454] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:22.312235 12915 leader_election.cc:302] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:22.312494 13709 raft_consensus.cc:2781] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.312556 13709 raft_consensus.cc:482] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.312582 13709 raft_consensus.cc:3037] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.313421 13709 raft_consensus.cc:504] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.313741 13709 leader_election.cc:288] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:22.314025 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca34c8d867f4f2293ad5aac5fb8aadb" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:22.314081 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca34c8d867f4f2293ad5aac5fb8aadb" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:22.314131 12587 raft_consensus.cc:3037] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.314193 12716 raft_consensus.cc:3037] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.314882 12587 raft_consensus.cc:2454] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:22.315003 12716 raft_consensus.cc:2454] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:22.315155 12915 leader_election.cc:302] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:22.315344 13709 raft_consensus.cc:2781] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.315433 13709 raft_consensus.cc:686] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:22.315740 13709 consensus_queue.cc:237] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.317948 12443 catalog_manager.cc:5484] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:22.334355 13718 raft_consensus.cc:482] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.334517 13718 raft_consensus.cc:504] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.335003 13718 leader_election.cc:288] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.335453 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbeba1e7c473403da87b5a0cd3c70d7e" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:22.335485 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbeba1e7c473403da87b5a0cd3c70d7e" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:22.335610 12848 raft_consensus.cc:2454] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:22.335610 12980 raft_consensus.cc:2454] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:22.335889 12523 leader_election.cc:302] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:22.336086 13718 raft_consensus.cc:2781] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.336156 13718 raft_consensus.cc:482] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.336210 13718 raft_consensus.cc:3037] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.337152 13718 raft_consensus.cc:504] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.337461 13718 leader_election.cc:288] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.337798 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbeba1e7c473403da87b5a0cd3c70d7e" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:22.337827 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbeba1e7c473403da87b5a0cd3c70d7e" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:22.337931 12848 raft_consensus.cc:3037] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.337957 12980 raft_consensus.cc:3037] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.338716 12980 raft_consensus.cc:2454] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:22.338840 12848 raft_consensus.cc:2454] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:22.339499 12522 leader_election.cc:302] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:22.339674 13718 raft_consensus.cc:2781] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.339761 13718 raft_consensus.cc:686] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:22.339901 13718 consensus_queue.cc:237] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.341495 12443 catalog_manager.cc:5484] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:22.453846 13751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:22.454085 13751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:22.457051 13751 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:09:22.546286 12980 raft_consensus.cc:1261] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.546941 13718 consensus_queue.cc:1035] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.547099 12848 raft_consensus.cc:1261] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.547549 13718 consensus_queue.cc:1035] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.548030 12587 raft_consensus.cc:1261] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.548040 12979 raft_consensus.cc:1261] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.549293 13744 consensus_queue.cc:1035] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.551065 12716 raft_consensus.cc:1261] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.551662 13709 consensus_queue.cc:1035] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.552153 12587 raft_consensus.cc:1261] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:22.552716 13744 consensus_queue.cc:1035] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:22.555774 13701 consensus_queue.cc:1035] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.213 ms
time per row: 0.0561064 ms
I20240501 14:09:22.609824 12437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:22.614910 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 7693e3e204634676a758726bb6d3d2ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.615020 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 7693e3e204634676a758726bb6d3d2ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.615216 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693e3e204634676a758726bb6d3d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.615293 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693e3e204634676a758726bb6d3d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.615636 12959 tablet_service.cc:1467] Processing CreateTablet for tablet e13e55ae52a949bc893de94082130157 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.615856 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13e55ae52a949bc893de94082130157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.616796 13731 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:22.617322 12960 tablet_service.cc:1467] Processing CreateTablet for tablet eb336cffd30b447b880af97da6882d27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.617358 12697 tablet_service.cc:1467] Processing CreateTablet for tablet e13e55ae52a949bc893de94082130157 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.617544 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336cffd30b447b880af97da6882d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.617640 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13e55ae52a949bc893de94082130157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.617641 13731 tablet_bootstrap.cc:654] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.618006 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 7693e3e204634676a758726bb6d3d2ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.618220 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693e3e204634676a758726bb6d3d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.619391 13729 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:22.619549 13728 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:22.619943 13730 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:22.620152 13729 tablet_bootstrap.cc:654] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.620332 13728 tablet_bootstrap.cc:654] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.620764 13730 tablet_bootstrap.cc:654] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.621285 13729 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:22.621348 13729 ts_tablet_manager.cc:1367] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:22.621726 13729 raft_consensus.cc:348] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.621758 12829 tablet_service.cc:1467] Processing CreateTablet for tablet eb336cffd30b447b880af97da6882d27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.621824 13729 raft_consensus.cc:374] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.621848 13729 raft_consensus.cc:729] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.621982 13729 consensus_queue.cc:260] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.622012 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336cffd30b447b880af97da6882d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.622203 13729 ts_tablet_manager.cc:1398] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.623183 13728 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:22.623257 13728 ts_tablet_manager.cc:1367] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:22.623690 13728 raft_consensus.cc:348] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.623783 13728 raft_consensus.cc:374] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.623814 13728 raft_consensus.cc:729] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.623821 12828 tablet_service.cc:1467] Processing CreateTablet for tablet e13e55ae52a949bc893de94082130157 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.623929 13728 consensus_queue.cc:260] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.624017 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13e55ae52a949bc893de94082130157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.624189 13728 ts_tablet_manager.cc:1398] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.624351 13728 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:22.625314 13728 tablet_bootstrap.cc:654] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.626861 13728 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:22.626938 13728 ts_tablet_manager.cc:1367] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.627451 12565 tablet_service.cc:1467] Processing CreateTablet for tablet eb336cffd30b447b880af97da6882d27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:22.627674 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336cffd30b447b880af97da6882d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:22.627761 13730 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:22.627835 13730 ts_tablet_manager.cc:1367] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:09:22.627913 13731 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:22.628000 13731 ts_tablet_manager.cc:1367] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 14:09:22.628006 13728 raft_consensus.cc:348] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.628126 13728 raft_consensus.cc:374] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.628158 13728 raft_consensus.cc:729] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.628264 13728 consensus_queue.cc:260] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.628229 13730 raft_consensus.cc:348] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.628342 13730 raft_consensus.cc:374] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.628372 13730 raft_consensus.cc:729] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.628449 13731 raft_consensus.cc:348] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.628518 13728 ts_tablet_manager.cc:1398] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:09:22.628486 13730 consensus_queue.cc:260] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.628540 13731 raft_consensus.cc:374] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.628568 13731 raft_consensus.cc:729] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.628685 13731 consensus_queue.cc:260] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.628744 13730 ts_tablet_manager.cc:1398] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:22.628916 13731 ts_tablet_manager.cc:1398] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.628926 13730 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:22.629063 13731 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:22.629823 13730 tablet_bootstrap.cc:654] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.629853 13731 tablet_bootstrap.cc:654] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.630751 13729 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:22.631639 13729 tablet_bootstrap.cc:654] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.631645 13731 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:22.631711 13731 ts_tablet_manager.cc:1367] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:22.632140 13731 raft_consensus.cc:348] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.632333 13731 raft_consensus.cc:374] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.632392 13731 raft_consensus.cc:729] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.632527 13731 consensus_queue.cc:260] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.633309 13731 ts_tablet_manager.cc:1398] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:22.633636 13731 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:22.634462 13731 tablet_bootstrap.cc:654] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:22.635533 13731 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:22.635610 13731 ts_tablet_manager.cc:1367] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:22.636022 13731 raft_consensus.cc:348] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.636122 13731 raft_consensus.cc:374] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.636154 13731 raft_consensus.cc:729] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.636266 13731 consensus_queue.cc:260] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.636447 13729 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:22.636521 13731 ts_tablet_manager.cc:1398] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.636533 13729 ts_tablet_manager.cc:1367] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:22.637005 13729 raft_consensus.cc:348] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.637109 13729 raft_consensus.cc:374] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.637151 13729 raft_consensus.cc:729] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.637284 13729 consensus_queue.cc:260] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.637527 13729 ts_tablet_manager.cc:1398] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.637806 13730 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:22.637885 13730 ts_tablet_manager.cc:1367] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:22.638314 13730 raft_consensus.cc:348] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.638412 13730 raft_consensus.cc:374] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:22.638506 13730 raft_consensus.cc:729] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:22.638625 13730 consensus_queue.cc:260] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.638922 13730 ts_tablet_manager.cc:1398] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:22.679400 13631 raft_consensus.cc:482] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.679524 13631 raft_consensus.cc:504] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.680006 13631 leader_election.cc:288] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:22.680416 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:22.680474 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:22.680574 12848 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:22.680620 12716 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:22.680877 12523 leader_election.cc:302] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:22.681030 13631 raft_consensus.cc:2781] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.681108 13631 raft_consensus.cc:482] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.681143 13631 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.681972 13631 raft_consensus.cc:504] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.682299 13631 leader_election.cc:288] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:22.682588 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:22.682711 12848 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.682660 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:22.682837 12716 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.683504 12848 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:22.683562 12716 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:22.683770 12523 leader_election.cc:302] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:22.683925 13631 raft_consensus.cc:2781] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.684006 13631 raft_consensus.cc:686] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:22.684167 13631 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:22.685705 12441 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:22.695210 13744 raft_consensus.cc:482] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.695318 13744 raft_consensus.cc:504] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.695749 13744 leader_election.cc:288] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.696110 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13e55ae52a949bc893de94082130157" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:22.696271 12980 raft_consensus.cc:2454] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:22.696559 12653 leader_election.cc:302] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:22.696729 13744 raft_consensus.cc:2781] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.696775 13744 raft_consensus.cc:482] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.696800 13744 raft_consensus.cc:3037] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.697786 13744 raft_consensus.cc:504] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.698143 13744 leader_election.cc:288] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.698511 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13e55ae52a949bc893de94082130157" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:22.698608 12980 raft_consensus.cc:3037] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.699542 12980 raft_consensus.cc:2454] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:22.699687 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13e55ae52a949bc893de94082130157" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:22.699723 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e13e55ae52a949bc893de94082130157" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:22.699817 12848 raft_consensus.cc:2454] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:22.699868 12653 leader_election.cc:302] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:22.700023 13744 raft_consensus.cc:2781] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.700124 13744 raft_consensus.cc:686] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:22.700271 13744 consensus_queue.cc:237] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:22.701805 12441 catalog_manager.cc:5484] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:22.786747 13631 raft_consensus.cc:482] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:22.786978 13631 raft_consensus.cc:504] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.787461 13631 leader_election.cc:288] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.787860 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336cffd30b447b880af97da6882d27" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:22.787943 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336cffd30b447b880af97da6882d27" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:22.788022 12849 raft_consensus.cc:2454] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:22.788079 12980 raft_consensus.cc:2454] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:22.788357 12522 leader_election.cc:302] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:22.788522 13631 raft_consensus.cc:2781] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:22.788587 13631 raft_consensus.cc:482] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:22.788625 13631 raft_consensus.cc:3037] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.789359 13631 raft_consensus.cc:504] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.789635 13631 leader_election.cc:288] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:22.789952 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336cffd30b447b880af97da6882d27" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:22.790048 12849 raft_consensus.cc:3037] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.790010 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336cffd30b447b880af97da6882d27" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:22.790194 12980 raft_consensus.cc:3037] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:22.790834 12849 raft_consensus.cc:2454] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:22.791111 12523 leader_election.cc:302] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:22.791147 12980 raft_consensus.cc:2454] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:22.791260 13631 raft_consensus.cc:2781] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:22.791361 13631 raft_consensus.cc:686] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:22.791504 13631 consensus_queue.cc:237] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:22.793190 12441 catalog_manager.cc:5484] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:22.930471 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:22.930703 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:22.933643 13804 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:09:23.007375 12980 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.008023 12848 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.008023 12849 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.008150 12716 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.008597 13718 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.008785 13631 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.008838 13582 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.009716 12980 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.010175 13695 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.014151 13582 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.015956 12847 raft_consensus.cc:3037] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.017050 12847 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.024641 13744 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.451 ms
time per row: 0.0527253 ms
I20240501 14:09:23.082896 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:23.088244 12565 tablet_service.cc:1467] Processing CreateTablet for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.088392 12828 tablet_service.cc:1467] Processing CreateTablet for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.088572 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.088690 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.089810 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.090076 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b152ca89c5a416f9b1a65217e4a950e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.090255 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 14f1774f12d34e91a9a5409a60ee1976 (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.090490 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f1774f12d34e91a9a5409a60ee1976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.090793 13731 tablet_bootstrap.cc:492] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:23.090976 12698 tablet_service.cc:1467] Processing CreateTablet for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.091220 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.091708 13731 tablet_bootstrap.cc:654] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.092298 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 14f1774f12d34e91a9a5409a60ee1976 (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.092523 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f1774f12d34e91a9a5409a60ee1976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.092747 13729 tablet_bootstrap.cc:492] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:23.093159 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 14f1774f12d34e91a9a5409a60ee1976 (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.093398 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14f1774f12d34e91a9a5409a60ee1976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.093559 13729 tablet_bootstrap.cc:654] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.094537 13729 tablet_bootstrap.cc:492] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:23.094570 12697 tablet_service.cc:1467] Processing CreateTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.094643 13729 ts_tablet_manager.cc:1367] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.095016 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e (DEFAULT_TABLE table=rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.095146 13729 raft_consensus.cc:348] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.095252 13731 tablet_bootstrap.cc:492] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:23.095245 13729 raft_consensus.cc:374] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.095292 13729 raft_consensus.cc:729] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.095325 13731 ts_tablet_manager.cc:1367] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:23.095414 13730 tablet_bootstrap.cc:492] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:23.095438 13729 consensus_queue.cc:260] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.095737 13729 ts_tablet_manager.cc:1398] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.095769 13731 raft_consensus.cc:348] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.095860 13731 raft_consensus.cc:374] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.095891 13731 raft_consensus.cc:729] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.095932 13729 tablet_bootstrap.cc:492] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:23.095250 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b152ca89c5a416f9b1a65217e4a950e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.096009 13731 consensus_queue.cc:260] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.096218 13730 tablet_bootstrap.cc:654] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.096297 13731 ts_tablet_manager.cc:1398] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.096913 13728 tablet_bootstrap.cc:492] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:23.097530 13731 tablet_bootstrap.cc:492] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:23.097656 13729 tablet_bootstrap.cc:654] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.097988 13728 tablet_bootstrap.cc:654] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.098652 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b152ca89c5a416f9b1a65217e4a950e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.099125 13728 tablet_bootstrap.cc:492] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:23.099201 13728 ts_tablet_manager.cc:1367] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.099622 13728 raft_consensus.cc:348] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.099776 13728 raft_consensus.cc:374] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.099814 13728 raft_consensus.cc:729] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.099936 13728 consensus_queue.cc:260] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.100208 13728 ts_tablet_manager.cc:1398] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.100881 13728 tablet_bootstrap.cc:492] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:23.101752 13728 tablet_bootstrap.cc:654] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.101902 13730 tablet_bootstrap.cc:492] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:23.101995 13730 ts_tablet_manager.cc:1367] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:23.102471 13730 raft_consensus.cc:348] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.102576 13730 raft_consensus.cc:374] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.102614 13730 raft_consensus.cc:729] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.102772 13730 consensus_queue.cc:260] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.103019 13730 ts_tablet_manager.cc:1398] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.103171 13730 tablet_bootstrap.cc:492] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:23.103935 13731 tablet_bootstrap.cc:654] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.103950 13730 tablet_bootstrap.cc:654] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.105188 13730 tablet_bootstrap.cc:492] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:23.105283 13730 ts_tablet_manager.cc:1367] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.105726 13728 tablet_bootstrap.cc:492] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:23.105715 13730 raft_consensus.cc:348] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.105805 13730 raft_consensus.cc:374] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.105803 13729 tablet_bootstrap.cc:492] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:23.105814 13728 ts_tablet_manager.cc:1367] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:23.105841 13730 raft_consensus.cc:729] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.105908 13729 ts_tablet_manager.cc:1367] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:23.105990 13730 consensus_queue.cc:260] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.106238 13730 ts_tablet_manager.cc:1398] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.106247 13728 raft_consensus.cc:348] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.106339 13728 raft_consensus.cc:374] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.106371 13728 raft_consensus.cc:729] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.106345 13729 raft_consensus.cc:348] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.106451 13729 raft_consensus.cc:374] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.106483 13729 raft_consensus.cc:729] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.106487 13728 consensus_queue.cc:260] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.106595 13729 consensus_queue.cc:260] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.106825 13728 ts_tablet_manager.cc:1398] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.106902 13729 ts_tablet_manager.cc:1398] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.107110 13729 tablet_bootstrap.cc:492] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:23.107446 13731 tablet_bootstrap.cc:492] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:23.107524 13731 ts_tablet_manager.cc:1367] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:23.108029 13729 tablet_bootstrap.cc:654] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.107976 13731 raft_consensus.cc:348] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.108093 13731 raft_consensus.cc:374] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.108131 13731 raft_consensus.cc:729] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.108289 13731 consensus_queue.cc:260] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.109369 13729 tablet_bootstrap.cc:492] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:23.109458 13729 ts_tablet_manager.cc:1367] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.109925 13729 raft_consensus.cc:348] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.110024 13729 raft_consensus.cc:374] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.110059 13729 raft_consensus.cc:729] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.110183 13729 consensus_queue.cc:260] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.110435 13729 ts_tablet_manager.cc:1398] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:23.110821 13731 ts_tablet_manager.cc:1398] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:23.162422 13834 raft_consensus.cc:482] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.162535 13834 raft_consensus.cc:504] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.162863 13834 leader_election.cc:288] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:23.163203 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b152ca89c5a416f9b1a65217e4a950e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:23.163300 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b152ca89c5a416f9b1a65217e4a950e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:23.163354 12979 raft_consensus.cc:2454] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:23.163446 12586 raft_consensus.cc:2454] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:23.163606 12653 leader_election.cc:302] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:23.163765 13834 raft_consensus.cc:2781] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.163831 13834 raft_consensus.cc:482] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.163875 13834 raft_consensus.cc:3037] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.164713 13834 raft_consensus.cc:504] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.165027 13834 leader_election.cc:288] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:23.165359 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b152ca89c5a416f9b1a65217e4a950e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:23.165392 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b152ca89c5a416f9b1a65217e4a950e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:23.165463 12979 raft_consensus.cc:3037] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.165498 12586 raft_consensus.cc:3037] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.166225 12979 raft_consensus.cc:2454] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:23.166504 12653 leader_election.cc:302] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:23.166683 13834 raft_consensus.cc:2781] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.166779 12586 raft_consensus.cc:2454] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:23.166791 13834 raft_consensus.cc:686] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:23.166944 13834 consensus_queue.cc:237] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.168586 12438 catalog_manager.cc:5484] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:23.228013 13709 raft_consensus.cc:482] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.228154 13709 raft_consensus.cc:504] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.228626 13709 leader_election.cc:288] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:23.229074 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f1774f12d34e91a9a5409a60ee1976" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:23.229225 12586 raft_consensus.cc:2454] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:23.229503 12915 leader_election.cc:302] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:23.229665 13709 raft_consensus.cc:2781] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.229743 13709 raft_consensus.cc:482] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.229776 13709 raft_consensus.cc:3037] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.231093 13709 raft_consensus.cc:504] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.231412 13709 leader_election.cc:288] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:23.231758 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f1774f12d34e91a9a5409a60ee1976" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:23.231884 12586 raft_consensus.cc:3037] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.232637 12586 raft_consensus.cc:2454] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:23.232945 12915 leader_election.cc:302] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:23.233116 13709 raft_consensus.cc:2781] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.233281 13709 raft_consensus.cc:686] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:23.233445 13709 consensus_queue.cc:237] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.233654 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f1774f12d34e91a9a5409a60ee1976" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:23.233790 12848 raft_consensus.cc:2454] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:23.233826 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14f1774f12d34e91a9a5409a60ee1976" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:23.233949 12847 raft_consensus.cc:3037] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.234846 12847 raft_consensus.cc:2454] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:23.235297 12438 catalog_manager.cc:5484] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:23.393357 13834 raft_consensus.cc:482] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.393497 13834 raft_consensus.cc:504] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.393924 13834 leader_election.cc:288] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:23.394318 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd40853ff8a42c4b5eaa3fa57bd0c3f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:23.394318 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd40853ff8a42c4b5eaa3fa57bd0c3f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:23.394475 12586 raft_consensus.cc:2454] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:23.394475 12847 raft_consensus.cc:2454] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:23.394811 12653 leader_election.cc:302] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:23.394992 13834 raft_consensus.cc:2781] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.395054 13834 raft_consensus.cc:482] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.395100 13834 raft_consensus.cc:3037] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.395897 13834 raft_consensus.cc:504] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.396245 13834 leader_election.cc:288] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:23.396575 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd40853ff8a42c4b5eaa3fa57bd0c3f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:23.396611 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdd40853ff8a42c4b5eaa3fa57bd0c3f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:23.396692 12586 raft_consensus.cc:3037] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.396721 12847 raft_consensus.cc:3037] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.397403 12847 raft_consensus.cc:2454] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:23.397626 12586 raft_consensus.cc:2454] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:23.397706 12654 leader_election.cc:302] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:23.397965 13834 raft_consensus.cc:2781] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.398039 13834 raft_consensus.cc:686] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:23.398185 13834 consensus_queue.cc:237] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:23.399821 12438 catalog_manager.cc:5484] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:23.582098 13855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:23.582347 13855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:23.585417 13855 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:09:23.675681 12847 raft_consensus.cc:1261] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.675681 12586 raft_consensus.cc:1261] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.676162 13852 consensus_queue.cc:1035] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.676327 13709 consensus_queue.cc:1035] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.680085 12586 raft_consensus.cc:1261] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.681085 13834 consensus_queue.cc:1035] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.682554 12847 raft_consensus.cc:1261] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.682966 13834 consensus_queue.cc:1035] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.685395 12585 raft_consensus.cc:1261] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.685814 13834 consensus_queue.cc:1035] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.688800 12979 raft_consensus.cc:1261] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:23.690210 13695 consensus_queue.cc:1035] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:23.698002 13888 mvcc.cc:205] Tried to move back new op lower bound from 7022889220859879424 to 7022889218733142016. Current Snapshot: MvccSnapshot[applied={T|T < 7022889220808237056}]
I20240501 14:09:23.718592 13900 mvcc.cc:205] Tried to move back new op lower bound from 7022889220852195328 to 7022889219680260096. Current Snapshot: MvccSnapshot[applied={T|T < 7022889220830044160}]
INSERT report
rows total: 2000
time total: 109.797 ms
time per row: 0.0548983 ms
I20240501 14:09:23.736711 12444 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:23.742810 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 727a51852ac2428eb23f7690fec56a16 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.742918 12697 tablet_service.cc:1467] Processing CreateTablet for tablet 727a51852ac2428eb23f7690fec56a16 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.743145 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727a51852ac2428eb23f7690fec56a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.743219 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727a51852ac2428eb23f7690fec56a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.743198 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 727a51852ac2428eb23f7690fec56a16 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.743471 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727a51852ac2428eb23f7690fec56a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.744666 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 7c778ae9911740c2b0cb42dfbc694bb1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.744904 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c778ae9911740c2b0cb42dfbc694bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.745644 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 63a76e89d58d48ba9d1b3b98ef6c172d (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.745863 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a76e89d58d48ba9d1b3b98ef6c172d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.745900 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 7c778ae9911740c2b0cb42dfbc694bb1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.746129 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c778ae9911740c2b0cb42dfbc694bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.746892 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 63a76e89d58d48ba9d1b3b98ef6c172d (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.746956 13915 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:23.747133 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a76e89d58d48ba9d1b3b98ef6c172d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.748023 13915 tablet_bootstrap.cc:654] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.748298 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 7c778ae9911740c2b0cb42dfbc694bb1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.748560 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c778ae9911740c2b0cb42dfbc694bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.748656 13916 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:23.748944 13913 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:23.749644 13913 tablet_bootstrap.cc:654] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.749778 13916 tablet_bootstrap.cc:654] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.750360 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 63a76e89d58d48ba9d1b3b98ef6c172d (DEFAULT_TABLE table=rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:23.750578 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a76e89d58d48ba9d1b3b98ef6c172d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:23.750854 13913 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:23.750875 13916 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:23.750963 13916 ts_tablet_manager.cc:1367] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.750998 13913 ts_tablet_manager.cc:1367] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:23.751447 13916 raft_consensus.cc:348] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.751564 13916 raft_consensus.cc:374] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.751602 13916 raft_consensus.cc:729] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.751480 13913 raft_consensus.cc:348] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.751721 13916 consensus_queue.cc:260] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.751762 13913 raft_consensus.cc:374] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.751869 13913 raft_consensus.cc:729] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.752012 13916 ts_tablet_manager.cc:1398] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.752054 13913 consensus_queue.cc:260] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.752184 13914 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:23.752472 13913 ts_tablet_manager.cc:1398] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.752950 13915 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:23.753041 13915 ts_tablet_manager.cc:1367] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:23.753268 13913 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:23.753501 13915 raft_consensus.cc:348] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.753615 13915 raft_consensus.cc:374] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.753652 13915 raft_consensus.cc:729] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.753787 13915 consensus_queue.cc:260] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.754112 13913 tablet_bootstrap.cc:654] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.754149 13915 ts_tablet_manager.cc:1398] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.754310 13915 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:23.755074 13915 tablet_bootstrap.cc:654] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.755064 13914 tablet_bootstrap.cc:654] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.756019 13915 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:23.756089 13914 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:23.756119 13915 ts_tablet_manager.cc:1367] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.756188 13914 ts_tablet_manager.cc:1367] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:23.756479 13915 raft_consensus.cc:348] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.756564 13915 raft_consensus.cc:374] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.756598 13915 raft_consensus.cc:729] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.756598 13914 raft_consensus.cc:348] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.756723 13914 raft_consensus.cc:374] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.756733 13915 consensus_queue.cc:260] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.756767 13914 raft_consensus.cc:729] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.757012 13915 ts_tablet_manager.cc:1398] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.757091 13914 consensus_queue.cc:260] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.757869 13913 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:23.757967 13914 ts_tablet_manager.cc:1398] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:23.757989 13913 ts_tablet_manager.cc:1367] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:23.758131 13914 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:23.758889 13913 raft_consensus.cc:348] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.759002 13913 raft_consensus.cc:374] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.759050 13913 raft_consensus.cc:729] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.759106 13914 tablet_bootstrap.cc:654] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.759212 13913 consensus_queue.cc:260] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.759518 13913 ts_tablet_manager.cc:1398] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.759745 13913 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:23.760597 13913 tablet_bootstrap.cc:654] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.760998 13914 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:23.761086 13914 ts_tablet_manager.cc:1367] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:23.761444 13914 raft_consensus.cc:348] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.761574 13914 raft_consensus.cc:374] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.761618 13914 raft_consensus.cc:729] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.761749 13914 consensus_queue.cc:260] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:23.762041 13914 ts_tablet_manager.cc:1398] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:23.762157 13913 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:23.762228 13913 ts_tablet_manager.cc:1367] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:23.762259 13914 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:23.762693 13698 raft_consensus.cc:482] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.762666 13913 raft_consensus.cc:348] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.762787 13913 raft_consensus.cc:374] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.762787 13698 raft_consensus.cc:504] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.762856 13913 raft_consensus.cc:729] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.762980 13913 consensus_queue.cc:260] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.763135 13914 tablet_bootstrap.cc:654] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:23.763195 13698 leader_election.cc:288] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:23.763629 13913 ts_tablet_manager.cc:1398] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:23.763653 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727a51852ac2428eb23f7690fec56a16" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:23.763764 12979 raft_consensus.cc:2454] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:23.763823 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727a51852ac2428eb23f7690fec56a16" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:23.763928 12718 raft_consensus.cc:2454] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:23.764187 12784 leader_election.cc:302] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:23.764430 13914 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:23.764518 13914 ts_tablet_manager.cc:1367] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:23.764906 13914 raft_consensus.cc:348] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.764952 13698 raft_consensus.cc:2781] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.765002 13914 raft_consensus.cc:374] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:23.765003 13698 raft_consensus.cc:482] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.765044 13698 raft_consensus.cc:3037] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.765036 13914 raft_consensus.cc:729] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:23.765168 13914 consensus_queue.cc:260] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.765417 13914 ts_tablet_manager.cc:1398] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:23.766036 13698 raft_consensus.cc:504] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.766384 13698 leader_election.cc:288] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:23.766682 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727a51852ac2428eb23f7690fec56a16" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:23.766816 12979 raft_consensus.cc:3037] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.766889 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727a51852ac2428eb23f7690fec56a16" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:23.766994 12718 raft_consensus.cc:3037] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.767553 12979 raft_consensus.cc:2454] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:23.767812 12784 leader_election.cc:302] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:23.767894 12718 raft_consensus.cc:2454] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:23.767944 13698 raft_consensus.cc:2781] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.768028 13698 raft_consensus.cc:686] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:23.768148 13698 consensus_queue.cc:237] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.769555 12437 catalog_manager.cc:5484] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:23.845557 13852 raft_consensus.cc:482] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:23.845712 13852 raft_consensus.cc:504] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.846163 13852 leader_election.cc:288] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:23.846640 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a76e89d58d48ba9d1b3b98ef6c172d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:23.846661 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a76e89d58d48ba9d1b3b98ef6c172d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:23.846794 12718 raft_consensus.cc:2454] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:23.846843 12847 raft_consensus.cc:2454] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:23.847086 12913 leader_election.cc:302] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:23.847270 13852 raft_consensus.cc:2781] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:23.847323 13852 raft_consensus.cc:482] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:23.847371 13852 raft_consensus.cc:3037] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.848124 13852 raft_consensus.cc:504] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.848419 13852 leader_election.cc:288] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:23.848748 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a76e89d58d48ba9d1b3b98ef6c172d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:23.848760 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a76e89d58d48ba9d1b3b98ef6c172d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:23.848843 12847 raft_consensus.cc:3037] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.848850 12718 raft_consensus.cc:3037] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:23.849537 12847 raft_consensus.cc:2454] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:23.849668 12718 raft_consensus.cc:2454] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:23.849845 12916 leader_election.cc:302] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:23.850054 13852 raft_consensus.cc:2781] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:23.850139 13852 raft_consensus.cc:686] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:23.850294 13852 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:23.852044 12437 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:24.041599 13698 raft_consensus.cc:482] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.041738 13698 raft_consensus.cc:504] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.042539 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c778ae9911740c2b0cb42dfbc694bb1" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:24.042539 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c778ae9911740c2b0cb42dfbc694bb1" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:24.042677 12586 raft_consensus.cc:2454] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:24.042677 12718 raft_consensus.cc:2454] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:24.043035 12782 leader_election.cc:302] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:24.043256 13647 raft_consensus.cc:2781] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.043347 13647 raft_consensus.cc:482] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.043380 13647 raft_consensus.cc:3037] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.044073 13698 leader_election.cc:288] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:24.046684 13647 raft_consensus.cc:504] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.047047 13647 leader_election.cc:288] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:24.047381 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c778ae9911740c2b0cb42dfbc694bb1" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:24.047510 12718 raft_consensus.cc:3037] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.048388 12718 raft_consensus.cc:2454] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:24.048628 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c778ae9911740c2b0cb42dfbc694bb1" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:24.048656 12782 leader_election.cc:302] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:24.048748 12586 raft_consensus.cc:3037] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.048815 13647 raft_consensus.cc:2781] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.048887 13647 raft_consensus.cc:686] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:24.049015 13647 consensus_queue.cc:237] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.049605 12586 raft_consensus.cc:2454] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:24.050791 12437 catalog_manager.cc:5484] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:24.218997 13929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:24.219245 13929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:24.222690 13929 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:24.235863 13647 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.241216 13943 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240501 14:09:24.297951 12586 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.298424 13943 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.299166 12718 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.301736 12847 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.303040 13852 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.303853 12716 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.304066 13647 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.306555 13852 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.321388 13946 mvcc.cc:205] Tried to move back new op lower bound from 7022889223359238144 to 7022889222346080256. Current Snapshot: MvccSnapshot[applied={T|T < 7022889223359238144}]
I20240501 14:09:24.333549 13942 mvcc.cc:205] Tried to move back new op lower bound from 7022889223366668288 to 7022889221531963392. Current Snapshot: MvccSnapshot[applied={T|T < 7022889223366668288}]
INSERT report
rows total: 2000
time total: 92.0277 ms
time per row: 0.0460139 ms
I20240501 14:09:24.360234 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:24.365706 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.365998 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea1af845a8e4f39a973d1f8ef691762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.366387 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.366703 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea1af845a8e4f39a973d1f8ef691762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.366667 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 5e22abf2e2e544eb9ac8dccca71117bc (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.367002 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e22abf2e2e544eb9ac8dccca71117bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.369760 12696 tablet_service.cc:1467] Processing CreateTablet for tablet c7fc40ff90834e84a781cd08a45a33f8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.369762 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.370000 13989 tablet_bootstrap.cc:492] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:24.370059 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7fc40ff90834e84a781cd08a45a33f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.370220 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea1af845a8e4f39a973d1f8ef691762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.370975 13989 tablet_bootstrap.cc:654] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.371126 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 5e22abf2e2e544eb9ac8dccca71117bc (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.371390 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e22abf2e2e544eb9ac8dccca71117bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.371721 12960 tablet_service.cc:1467] Processing CreateTablet for tablet c7fc40ff90834e84a781cd08a45a33f8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.371778 12565 tablet_service.cc:1467] Processing CreateTablet for tablet c7fc40ff90834e84a781cd08a45a33f8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.371951 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7fc40ff90834e84a781cd08a45a33f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.371953 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7fc40ff90834e84a781cd08a45a33f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.374174 13991 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:24.375178 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 5e22abf2e2e544eb9ac8dccca71117bc (DEFAULT_TABLE table=rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.375416 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e22abf2e2e544eb9ac8dccca71117bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.375412 13989 tablet_bootstrap.cc:492] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:24.375512 13989 ts_tablet_manager.cc:1367] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:24.375981 13989 raft_consensus.cc:348] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.376102 13989 raft_consensus.cc:374] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.376163 13989 raft_consensus.cc:729] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.376296 13989 consensus_queue.cc:260] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.376645 13989 ts_tablet_manager.cc:1398] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.376842 13991 tablet_bootstrap.cc:654] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.377529 13990 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:24.378477 13990 tablet_bootstrap.cc:654] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.379521 13990 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:24.379606 13990 ts_tablet_manager.cc:1367] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:24.379621 13991 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:24.379719 13991 ts_tablet_manager.cc:1367] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:24.379979 13990 raft_consensus.cc:348] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.380065 13990 raft_consensus.cc:374] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.380097 13990 raft_consensus.cc:729] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.380096 13991 raft_consensus.cc:348] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.380218 13991 raft_consensus.cc:374] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.380255 13991 raft_consensus.cc:729] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.380255 13990 consensus_queue.cc:260] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.380391 13991 consensus_queue.cc:260] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.380535 13990 ts_tablet_manager.cc:1398] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.380671 13991 ts_tablet_manager.cc:1398] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.380751 13990 tablet_bootstrap.cc:492] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:24.380844 13991 tablet_bootstrap.cc:492] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:24.381644 13991 tablet_bootstrap.cc:654] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.381644 13990 tablet_bootstrap.cc:654] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.384701 13997 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:24.386313 13991 tablet_bootstrap.cc:492] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:24.386399 13991 ts_tablet_manager.cc:1367] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:24.386853 13997 tablet_bootstrap.cc:654] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.386854 13991 raft_consensus.cc:348] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.386945 13991 raft_consensus.cc:374] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.386976 13991 raft_consensus.cc:729] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.387120 13991 consensus_queue.cc:260] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.387876 13997 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:24.387873 13991 ts_tablet_manager.cc:1398] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.387956 13997 ts_tablet_manager.cc:1367] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:24.388279 13997 raft_consensus.cc:348] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.388383 13997 raft_consensus.cc:374] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.388407 13997 raft_consensus.cc:729] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.388517 13997 consensus_queue.cc:260] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.388636 13990 tablet_bootstrap.cc:492] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:24.388715 13990 ts_tablet_manager.cc:1367] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:24.388739 13997 ts_tablet_manager.cc:1398] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:24.388870 13997 tablet_bootstrap.cc:492] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:24.389513 13997 tablet_bootstrap.cc:654] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.389528 13990 raft_consensus.cc:348] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.389638 13990 raft_consensus.cc:374] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.389670 13990 raft_consensus.cc:729] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.389788 13990 consensus_queue.cc:260] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.390599 13990 ts_tablet_manager.cc:1398] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:24.390815 13990 tablet_bootstrap.cc:492] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:24.391667 13997 tablet_bootstrap.cc:492] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:24.391741 13997 ts_tablet_manager.cc:1367] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:24.391783 13990 tablet_bootstrap.cc:654] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.392112 13997 raft_consensus.cc:348] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.392205 13997 raft_consensus.cc:374] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.392231 13997 raft_consensus.cc:729] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.392465 13997 consensus_queue.cc:260] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.393362 13990 tablet_bootstrap.cc:492] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:24.393433 13990 ts_tablet_manager.cc:1367] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.393867 13990 raft_consensus.cc:348] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.393991 13990 raft_consensus.cc:374] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.394057 13990 raft_consensus.cc:729] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.394184 13990 consensus_queue.cc:260] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.394469 13990 ts_tablet_manager.cc:1398] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.394793 13997 ts_tablet_manager.cc:1398] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:24.396709 13997 tablet_bootstrap.cc:492] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:24.397925 13997 tablet_bootstrap.cc:654] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.399065 13997 tablet_bootstrap.cc:492] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:24.399147 13997 ts_tablet_manager.cc:1367] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.399562 13997 raft_consensus.cc:348] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.399679 13997 raft_consensus.cc:374] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.399749 13997 raft_consensus.cc:729] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.399876 13997 consensus_queue.cc:260] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.400200 13997 ts_tablet_manager.cc:1398] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.424247 13930 raft_consensus.cc:482] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.424376 13930 raft_consensus.cc:504] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.424775 13930 leader_election.cc:288] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:24.425210 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e22abf2e2e544eb9ac8dccca71117bc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:24.425217 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e22abf2e2e544eb9ac8dccca71117bc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:24.425355 12847 raft_consensus.cc:2454] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:24.425359 12979 raft_consensus.cc:2454] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:24.425662 12522 leader_election.cc:302] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:24.425837 13930 raft_consensus.cc:2781] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.425895 13930 raft_consensus.cc:482] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.425949 13930 raft_consensus.cc:3037] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.426877 13930 raft_consensus.cc:504] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.427201 13930 leader_election.cc:288] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:24.427495 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e22abf2e2e544eb9ac8dccca71117bc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:24.427613 12847 raft_consensus.cc:3037] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.427578 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e22abf2e2e544eb9ac8dccca71117bc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:24.427693 12979 raft_consensus.cc:3037] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.428534 12979 raft_consensus.cc:2454] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:24.428534 12847 raft_consensus.cc:2454] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:24.428817 12522 leader_election.cc:302] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:24.428966 13930 raft_consensus.cc:2781] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.429073 13930 raft_consensus.cc:686] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:24.429212 13930 consensus_queue.cc:237] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.430974 12445 catalog_manager.cc:5484] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:24.475756 13930 raft_consensus.cc:482] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.475875 13930 raft_consensus.cc:504] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.476297 13930 leader_election.cc:288] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:24.476636 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fc40ff90834e84a781cd08a45a33f8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:24.476698 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fc40ff90834e84a781cd08a45a33f8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:24.476781 12979 raft_consensus.cc:2454] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:24.476831 12716 raft_consensus.cc:2454] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:24.477066 12522 leader_election.cc:302] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:24.477217 13930 raft_consensus.cc:2781] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.477278 13930 raft_consensus.cc:482] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.477310 13930 raft_consensus.cc:3037] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.478044 13930 raft_consensus.cc:504] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.478317 13930 leader_election.cc:288] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:24.478670 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fc40ff90834e84a781cd08a45a33f8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:24.478668 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7fc40ff90834e84a781cd08a45a33f8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:24.478813 12979 raft_consensus.cc:3037] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.478826 12716 raft_consensus.cc:3037] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.479722 12979 raft_consensus.cc:2454] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:24.479722 12716 raft_consensus.cc:2454] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:24.480007 12522 leader_election.cc:302] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:24.480181 13930 raft_consensus.cc:2781] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.480264 13930 raft_consensus.cc:686] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:24.480386 13930 consensus_queue.cc:237] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.482071 12445 catalog_manager.cc:5484] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:24.511929 13987 raft_consensus.cc:482] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.512097 13987 raft_consensus.cc:504] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.512859 13987 leader_election.cc:288] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:24.513309 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea1af845a8e4f39a973d1f8ef691762" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:24.513315 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea1af845a8e4f39a973d1f8ef691762" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:24.513453 12716 raft_consensus.cc:2454] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:24.513475 12586 raft_consensus.cc:2454] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:24.513756 12913 leader_election.cc:302] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:24.513938 13987 raft_consensus.cc:2781] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.513998 13987 raft_consensus.cc:482] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.514050 13987 raft_consensus.cc:3037] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.514955 13987 raft_consensus.cc:504] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.515281 13987 leader_election.cc:288] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:24.515625 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea1af845a8e4f39a973d1f8ef691762" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:24.515631 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea1af845a8e4f39a973d1f8ef691762" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:24.515722 12716 raft_consensus.cc:3037] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.515756 12586 raft_consensus.cc:3037] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.516422 12716 raft_consensus.cc:2454] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:24.516505 12586 raft_consensus.cc:2454] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:24.516676 12913 leader_election.cc:302] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:24.516827 13987 raft_consensus.cc:2781] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.516916 13987 raft_consensus.cc:686] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:24.517021 13987 consensus_queue.cc:237] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.518723 12445 catalog_manager.cc:5484] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:24.636370 14005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:24.636647 14005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:24.639894 14005 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 14:09:24.730304 12716 raft_consensus.cc:1261] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.730499 12586 raft_consensus.cc:1261] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.730899 13987 consensus_queue.cc:1035] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.733788 12979 raft_consensus.cc:1261] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.733798 12847 raft_consensus.cc:1261] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.733968 13987 consensus_queue.cc:1035] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.734635 13930 consensus_queue.cc:1035] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.736526 12718 raft_consensus.cc:1261] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.736953 13718 consensus_queue.cc:1035] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.741196 13930 consensus_queue.cc:1035] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:24.742717 12979 raft_consensus.cc:1261] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:24.744480 13718 consensus_queue.cc:1035] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.735 ms
time per row: 0.0583676 ms
I20240501 14:09:24.796996 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:24.802232 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.802232 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.802331 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.802527 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.802527 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.802609 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.804289 13991 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:24.804508 13989 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:24.804558 13997 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:24.805123 13991 tablet_bootstrap.cc:654] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.805222 13989 tablet_bootstrap.cc:654] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.805295 13997 tablet_bootstrap.cc:654] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.806134 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 5648b7e0a1e744cea470feb7911d3d2c (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.806365 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648b7e0a1e744cea470feb7911d3d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.806775 13989 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:24.806762 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 5648b7e0a1e744cea470feb7911d3d2c (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.806844 13989 ts_tablet_manager.cc:1367] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:24.806819 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 16324878552c4b6dbf10e8d5f73a0436 (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.806999 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648b7e0a1e744cea470feb7911d3d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.807188 13989 raft_consensus.cc:348] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.807298 13989 raft_consensus.cc:374] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.807320 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16324878552c4b6dbf10e8d5f73a0436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.807327 13989 raft_consensus.cc:729] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.807430 13989 consensus_queue.cc:260] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.807672 13989 ts_tablet_manager.cc:1398] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.808228 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 16324878552c4b6dbf10e8d5f73a0436 (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.808439 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16324878552c4b6dbf10e8d5f73a0436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.808701 13991 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:24.808779 13991 ts_tablet_manager.cc:1367] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:24.809206 13991 raft_consensus.cc:348] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.809286 13991 raft_consensus.cc:374] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.809306 13991 raft_consensus.cc:729] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.809406 13991 consensus_queue.cc:260] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.809643 13991 ts_tablet_manager.cc:1398] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.809804 13990 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:24.809834 13997 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:24.809912 13997 ts_tablet_manager.cc:1367] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:24.810349 13997 raft_consensus.cc:348] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.810422 13990 tablet_bootstrap.cc:654] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.810444 13997 raft_consensus.cc:374] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.810474 13997 raft_consensus.cc:729] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.810585 13997 consensus_queue.cc:260] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.810917 13997 ts_tablet_manager.cc:1398] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.811345 13989 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:24.811300 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 16324878552c4b6dbf10e8d5f73a0436 (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.811522 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16324878552c4b6dbf10e8d5f73a0436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.812196 13989 tablet_bootstrap.cc:654] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.813238 13989 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:24.813344 13989 ts_tablet_manager.cc:1367] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:24.813694 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 5648b7e0a1e744cea470feb7911d3d2c (DEFAULT_TABLE table=rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:24.813836 13989 raft_consensus.cc:348] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.813938 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648b7e0a1e744cea470feb7911d3d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:24.813948 13989 raft_consensus.cc:374] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.813987 13989 raft_consensus.cc:729] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.814116 13989 consensus_queue.cc:260] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.814462 13989 ts_tablet_manager.cc:1398] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.814843 13989 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:24.815593 13989 tablet_bootstrap.cc:654] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.816516 13997 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:24.816637 13990 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:24.816709 13990 ts_tablet_manager.cc:1367] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:24.816751 13991 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:24.817150 13990 raft_consensus.cc:348] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.817250 13990 raft_consensus.cc:374] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.817291 13990 raft_consensus.cc:729] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.817445 13990 consensus_queue.cc:260] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.817696 13990 ts_tablet_manager.cc:1398] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:24.817811 13991 tablet_bootstrap.cc:654] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.817821 13997 tablet_bootstrap.cc:654] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.818642 13989 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:24.818719 13989 ts_tablet_manager.cc:1367] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:24.819177 13997 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:24.819156 13989 raft_consensus.cc:348] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.819245 13989 raft_consensus.cc:374] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.819265 13997 ts_tablet_manager.cc:1367] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.819269 13991 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:24.819295 13989 raft_consensus.cc:729] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.819344 13991 ts_tablet_manager.cc:1367] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:24.819420 13989 consensus_queue.cc:260] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.819686 13989 ts_tablet_manager.cc:1398] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:24.819661 13997 raft_consensus.cc:348] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.819754 13997 raft_consensus.cc:374] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.819723 13991 raft_consensus.cc:348] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.819806 13997 raft_consensus.cc:729] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.819845 13991 raft_consensus.cc:374] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.819888 13991 raft_consensus.cc:729] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.819941 13997 consensus_queue.cc:260] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.819998 13991 consensus_queue.cc:260] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.820223 13991 ts_tablet_manager.cc:1398] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.820233 13997 ts_tablet_manager.cc:1398] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:24.820775 13990 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:24.821624 13990 tablet_bootstrap.cc:654] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:24.822975 13990 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:24.823065 13990 ts_tablet_manager.cc:1367] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:24.823503 13990 raft_consensus.cc:348] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.823597 13990 raft_consensus.cc:374] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:24.823640 13990 raft_consensus.cc:729] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:24.823745 13990 consensus_queue.cc:260] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:24.824030 13990 ts_tablet_manager.cc:1398] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:24.845412 13969 raft_consensus.cc:482] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.845499 13969 raft_consensus.cc:504] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.845850 13969 leader_election.cc:288] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:24.846122 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5648b7e0a1e744cea470feb7911d3d2c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:24.846246 12979 raft_consensus.cc:2454] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:24.846237 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5648b7e0a1e744cea470feb7911d3d2c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:24.846393 12586 raft_consensus.cc:2454] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:24.846534 12784 leader_election.cc:302] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:24.846689 13969 raft_consensus.cc:2781] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.846765 13969 raft_consensus.cc:482] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.846809 13969 raft_consensus.cc:3037] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.847519 13969 raft_consensus.cc:504] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.847769 13969 leader_election.cc:288] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:24.848027 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5648b7e0a1e744cea470feb7911d3d2c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:24.848095 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5648b7e0a1e744cea470feb7911d3d2c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:24.848137 12979 raft_consensus.cc:3037] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.848196 12586 raft_consensus.cc:3037] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.848888 12586 raft_consensus.cc:2454] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:24.848901 12979 raft_consensus.cc:2454] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:24.849160 12784 leader_election.cc:302] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:24.849334 13969 raft_consensus.cc:2781] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.849406 13969 raft_consensus.cc:686] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:24.849531 13969 consensus_queue.cc:237] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:24.851084 12444 catalog_manager.cc:5484] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:24.866571 13979 raft_consensus.cc:482] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:24.866698 13979 raft_consensus.cc:504] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.867130 13979 leader_election.cc:288] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:24.867523 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:24.867556 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:24.867660 12847 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:24.867673 12586 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:24.867942 12654 leader_election.cc:302] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:24.868093 13979 raft_consensus.cc:2781] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:24.868150 13979 raft_consensus.cc:482] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:24.868201 13979 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.869010 13979 raft_consensus.cc:504] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.869328 13979 leader_election.cc:288] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:24.869596 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:24.869614 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:24.869697 12586 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.869714 12847 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:24.870378 12586 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:24.870558 12847 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:24.870656 12653 leader_election.cc:302] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:24.870893 13979 raft_consensus.cc:2781] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:24.870970 13979 raft_consensus.cc:686] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:24.871115 13979 consensus_queue.cc:237] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:24.872893 12444 catalog_manager.cc:5484] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:25.065234 13979 raft_consensus.cc:482] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.065364 13979 raft_consensus.cc:504] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.065766 13979 leader_election.cc:288] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:25.066161 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16324878552c4b6dbf10e8d5f73a0436" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:25.066176 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16324878552c4b6dbf10e8d5f73a0436" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:25.066282 12847 raft_consensus.cc:2454] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:25.066318 12979 raft_consensus.cc:2454] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:25.066570 12654 leader_election.cc:302] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:25.066752 13979 raft_consensus.cc:2781] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.066810 13979 raft_consensus.cc:482] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.066845 13979 raft_consensus.cc:3037] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.067564 13979 raft_consensus.cc:504] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.067840 13979 leader_election.cc:288] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:25.068130 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16324878552c4b6dbf10e8d5f73a0436" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:25.068133 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16324878552c4b6dbf10e8d5f73a0436" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:25.068240 12847 raft_consensus.cc:3037] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.068240 12979 raft_consensus.cc:3037] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.069116 12979 raft_consensus.cc:2454] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:25.069116 12847 raft_consensus.cc:2454] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:25.069392 12653 leader_election.cc:302] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:25.069599 13979 raft_consensus.cc:2781] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.069720 13979 raft_consensus.cc:686] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:25.069875 13979 consensus_queue.cc:237] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.071537 12444 catalog_manager.cc:5484] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:25.208370 14060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:25.208590 14060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:25.211732 14060 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:09:25.232651 13969 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.237411 13969 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 14:09:25.306828 12587 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.310640 12848 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.310873 12847 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.311106 13695 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.311424 13695 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.312996 13979 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.313166 12978 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.314069 13979 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.49 ms
time per row: 0.0577449 ms
I20240501 14:09:25.376811 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:25.382326 12829 tablet_service.cc:1467] Processing CreateTablet for tablet afe9066fbac74c29b51fce22a672b65b (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.382448 12698 tablet_service.cc:1467] Processing CreateTablet for tablet afe9066fbac74c29b51fce22a672b65b (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.382639 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe9066fbac74c29b51fce22a672b65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.382761 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe9066fbac74c29b51fce22a672b65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.382845 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 59c4576fd61a4209ab979f8d35243e94 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.383045 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c4576fd61a4209ab979f8d35243e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.384552 12567 tablet_service.cc:1467] Processing CreateTablet for tablet afe9066fbac74c29b51fce22a672b65b (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.384699 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 59c4576fd61a4209ab979f8d35243e94 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.384840 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe9066fbac74c29b51fce22a672b65b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.385007 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c4576fd61a4209ab979f8d35243e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.385097 14118 tablet_bootstrap.cc:492] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:25.386492 14118 tablet_bootstrap.cc:654] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.386945 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 59c4576fd61a4209ab979f8d35243e94 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.386991 12959 tablet_service.cc:1467] Processing CreateTablet for tablet b547fcb34f12402abccdb23678c95e26 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.387137 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c4576fd61a4209ab979f8d35243e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.387181 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547fcb34f12402abccdb23678c95e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.388634 14121 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:25.388958 12566 tablet_service.cc:1467] Processing CreateTablet for tablet b547fcb34f12402abccdb23678c95e26 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.389170 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547fcb34f12402abccdb23678c95e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.389549 14121 tablet_bootstrap.cc:654] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.390578 14121 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:25.390659 14121 ts_tablet_manager.cc:1367] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:25.391003 14118 tablet_bootstrap.cc:492] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:25.391076 14118 ts_tablet_manager.cc:1367] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:25.391120 14121 raft_consensus.cc:348] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.391220 14121 raft_consensus.cc:374] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.391248 14121 raft_consensus.cc:729] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.391372 14121 consensus_queue.cc:260] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.391502 14118 raft_consensus.cc:348] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.391587 14118 raft_consensus.cc:374] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.391618 14118 raft_consensus.cc:729] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.391642 14121 ts_tablet_manager.cc:1398] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.391727 14118 consensus_queue.cc:260] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.391793 14121 tablet_bootstrap.cc:492] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:25.391996 14118 ts_tablet_manager.cc:1398] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:25.392164 14118 tablet_bootstrap.cc:492] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:25.392653 14121 tablet_bootstrap.cc:654] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.393049 14118 tablet_bootstrap.cc:654] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.393234 12829 tablet_service.cc:1467] Processing CreateTablet for tablet b547fcb34f12402abccdb23678c95e26 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.393642 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547fcb34f12402abccdb23678c95e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.394127 14121 tablet_bootstrap.cc:492] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:25.394201 14121 ts_tablet_manager.cc:1367] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.394563 14121 raft_consensus.cc:348] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.394654 14121 raft_consensus.cc:374] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.394690 14121 raft_consensus.cc:729] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.394877 14121 consensus_queue.cc:260] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.395179 14121 ts_tablet_manager.cc:1398] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:25.396755 14119 tablet_bootstrap.cc:492] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:25.398054 14119 tablet_bootstrap.cc:654] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.399147 14117 tablet_bootstrap.cc:492] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:25.399950 14119 tablet_bootstrap.cc:492] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:25.400004 14118 tablet_bootstrap.cc:492] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:25.400038 14119 ts_tablet_manager.cc:1367] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:25.400095 14118 ts_tablet_manager.cc:1367] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 14:09:25.400463 14118 raft_consensus.cc:348] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.400561 14118 raft_consensus.cc:374] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.400482 14119 raft_consensus.cc:348] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.400600 14118 raft_consensus.cc:729] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.400632 14119 raft_consensus.cc:374] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.400727 14119 raft_consensus.cc:729] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.400739 14118 consensus_queue.cc:260] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.400869 14119 consensus_queue.cc:260] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.401015 14118 ts_tablet_manager.cc:1398] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:25.401167 14119 ts_tablet_manager.cc:1398] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.401338 14119 tablet_bootstrap.cc:492] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:25.402060 14117 tablet_bootstrap.cc:654] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.402122 14119 tablet_bootstrap.cc:654] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.403954 14119 tablet_bootstrap.cc:492] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:25.404076 14119 ts_tablet_manager.cc:1367] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:25.404517 14119 raft_consensus.cc:348] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.404736 14119 raft_consensus.cc:374] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.404803 14119 raft_consensus.cc:729] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.404950 14119 consensus_queue.cc:260] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.405252 14119 ts_tablet_manager.cc:1398] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.405470 14119 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:25.406289 14119 tablet_bootstrap.cc:654] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.407337 14119 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:25.407410 14119 ts_tablet_manager.cc:1367] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:25.407465 14117 tablet_bootstrap.cc:492] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:25.407568 14117 ts_tablet_manager.cc:1367] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:25.407927 14119 raft_consensus.cc:348] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.408015 14119 raft_consensus.cc:374] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.408041 14119 raft_consensus.cc:729] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.407999 14117 raft_consensus.cc:348] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.408094 14117 raft_consensus.cc:374] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.408114 14117 raft_consensus.cc:729] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.408151 14119 consensus_queue.cc:260] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.408205 14117 consensus_queue.cc:260] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.408380 14119 ts_tablet_manager.cc:1398] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:25.408440 14117 ts_tablet_manager.cc:1398] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:25.408598 14117 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:25.409482 14117 tablet_bootstrap.cc:654] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.415447 14117 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:25.415552 14117 ts_tablet_manager.cc:1367] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:25.416033 14117 raft_consensus.cc:348] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.416131 14117 raft_consensus.cc:374] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.416152 14117 raft_consensus.cc:729] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.416282 14117 consensus_queue.cc:260] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.416566 14117 ts_tablet_manager.cc:1398] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:25.555747 13718 raft_consensus.cc:482] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.555862 13718 raft_consensus.cc:504] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.556324 13718 leader_election.cc:288] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:25.556669 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547fcb34f12402abccdb23678c95e26" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:25.556833 12847 raft_consensus.cc:2454] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:25.556797 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547fcb34f12402abccdb23678c95e26" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:25.556946 12978 raft_consensus.cc:2454] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:25.557130 12523 leader_election.cc:302] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:25.557271 13718 raft_consensus.cc:2781] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.557340 13718 raft_consensus.cc:482] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.557379 13718 raft_consensus.cc:3037] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.558207 13718 raft_consensus.cc:504] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.558538 13718 leader_election.cc:288] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:25.558842 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547fcb34f12402abccdb23678c95e26" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:25.558898 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b547fcb34f12402abccdb23678c95e26" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:25.558946 12847 raft_consensus.cc:3037] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.559047 12978 raft_consensus.cc:3037] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.559664 12847 raft_consensus.cc:2454] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:25.559854 12978 raft_consensus.cc:2454] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:25.559939 12523 leader_election.cc:302] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:25.560133 13718 raft_consensus.cc:2781] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.560307 13718 raft_consensus.cc:686] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:25.560446 13718 consensus_queue.cc:237] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.562083 12445 catalog_manager.cc:5484] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:25.568797 13718 raft_consensus.cc:482] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.568877 13718 raft_consensus.cc:504] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.569196 13718 leader_election.cc:288] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:25.569486 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4576fd61a4209ab979f8d35243e94" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:25.569624 12978 raft_consensus.cc:2454] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:25.569612 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4576fd61a4209ab979f8d35243e94" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:25.569757 12716 raft_consensus.cc:2454] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:25.569895 12522 leader_election.cc:302] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:25.570060 13718 raft_consensus.cc:2781] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.570107 13718 raft_consensus.cc:482] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.570134 13718 raft_consensus.cc:3037] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.570922 13718 raft_consensus.cc:504] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.571240 13718 leader_election.cc:288] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:25.571522 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4576fd61a4209ab979f8d35243e94" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:25.571545 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4576fd61a4209ab979f8d35243e94" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:25.571637 12716 raft_consensus.cc:3037] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.571637 12978 raft_consensus.cc:3037] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.572539 12716 raft_consensus.cc:2454] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:25.572538 12978 raft_consensus.cc:2454] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:25.572806 12522 leader_election.cc:302] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:25.572968 13718 raft_consensus.cc:2781] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.573055 13718 raft_consensus.cc:686] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:25.573179 13718 consensus_queue.cc:237] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.574915 12445 catalog_manager.cc:5484] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:25.648463 13718 raft_consensus.cc:482] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.648589 13718 raft_consensus.cc:504] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.648957 13718 leader_election.cc:288] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:25.649331 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe9066fbac74c29b51fce22a672b65b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:25.649330 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe9066fbac74c29b51fce22a672b65b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:25.649471 12847 raft_consensus.cc:2454] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:25.649471 12716 raft_consensus.cc:2454] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:25.649739 12523 leader_election.cc:302] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:25.649947 13718 raft_consensus.cc:2781] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:25.650003 13718 raft_consensus.cc:482] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:25.650041 13718 raft_consensus.cc:3037] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.650904 13718 raft_consensus.cc:504] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.651190 13718 leader_election.cc:288] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:25.651475 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe9066fbac74c29b51fce22a672b65b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:25.651584 12847 raft_consensus.cc:3037] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.651631 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe9066fbac74c29b51fce22a672b65b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:25.651762 12716 raft_consensus.cc:3037] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:25.652376 12847 raft_consensus.cc:2454] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:25.652544 12716 raft_consensus.cc:2454] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:25.652655 12523 leader_election.cc:302] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:25.652830 13718 raft_consensus.cc:2781] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:25.652927 13718 raft_consensus.cc:686] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:25.653064 13718 consensus_queue.cc:237] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:25.654695 12445 catalog_manager.cc:5484] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:25.786847 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:25.787103 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:25.790303 14134 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:09:25.881682 12978 raft_consensus.cc:1261] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.881867 12716 raft_consensus.cc:1261] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.882377 13718 consensus_queue.cc:1035] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.882517 14130 consensus_queue.cc:1035] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.883898 12847 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.884194 12848 raft_consensus.cc:1261] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.884718 12978 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.885094 12716 raft_consensus.cc:1261] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:25.886968 14130 consensus_queue.cc:1035] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.887025 13718 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.887312 14130 consensus_queue.cc:1035] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:25.887342 13718 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.285 ms
time per row: 0.0546424 ms
I20240501 14:09:25.942505 12445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:25.947989 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 8147fae0de98429ab6554c7e7f5bf8ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.948276 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8147fae0de98429ab6554c7e7f5bf8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.948191 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 8147fae0de98429ab6554c7e7f5bf8ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.948305 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 8147fae0de98429ab6554c7e7f5bf8ee (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.948500 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8147fae0de98429ab6554c7e7f5bf8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.948546 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8147fae0de98429ab6554c7e7f5bf8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.950620 14199 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:25.951220 12567 tablet_service.cc:1467] Processing CreateTablet for tablet d48da29c058b4db3a5b45b25755be1cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.951345 12698 tablet_service.cc:1467] Processing CreateTablet for tablet d48da29c058b4db3a5b45b25755be1cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.951504 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48da29c058b4db3a5b45b25755be1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.951553 14199 tablet_bootstrap.cc:654] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.951555 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48da29c058b4db3a5b45b25755be1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.951872 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 68b04992f1f94b36b01dfde24a42318d (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.952064 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b04992f1f94b36b01dfde24a42318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.952181 14198 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:25.953104 14198 tablet_bootstrap.cc:654] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.953784 14201 tablet_bootstrap.cc:492] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:25.954089 12828 tablet_service.cc:1467] Processing CreateTablet for tablet d48da29c058b4db3a5b45b25755be1cf (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.954319 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48da29c058b4db3a5b45b25755be1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.954562 14201 tablet_bootstrap.cc:654] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.954649 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 68b04992f1f94b36b01dfde24a42318d (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.954922 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b04992f1f94b36b01dfde24a42318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.955145 14197 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:25.955749 14197 tablet_bootstrap.cc:654] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.956530 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 68b04992f1f94b36b01dfde24a42318d (DEFAULT_TABLE table=rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:25.956715 14197 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:25.956745 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b04992f1f94b36b01dfde24a42318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:25.956789 14197 ts_tablet_manager.cc:1367] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.957020 14199 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:25.957091 14199 ts_tablet_manager.cc:1367] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:25.957257 14197 raft_consensus.cc:348] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.957386 14197 raft_consensus.cc:374] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.957459 14197 raft_consensus.cc:729] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.957520 14199 raft_consensus.cc:348] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.957612 14199 raft_consensus.cc:374] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.957643 14199 raft_consensus.cc:729] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.957659 14197 consensus_queue.cc:260] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.957753 14199 consensus_queue.cc:260] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.958009 14199 ts_tablet_manager.cc:1398] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.958009 14197 ts_tablet_manager.cc:1398] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.958171 14201 tablet_bootstrap.cc:492] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:25.958199 14197 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:25.958199 14199 tablet_bootstrap.cc:492] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:25.958252 14201 ts_tablet_manager.cc:1367] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:25.958738 14201 raft_consensus.cc:348] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:25.958884 14201 raft_consensus.cc:374] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.958914 14201 raft_consensus.cc:729] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.959097 14199 tablet_bootstrap.cc:654] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.959220 14197 tablet_bootstrap.cc:654] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.960157 14198 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:25.960248 14198 ts_tablet_manager.cc:1367] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:25.960639 14198 raft_consensus.cc:348] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.960738 14198 raft_consensus.cc:374] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.960765 14198 raft_consensus.cc:729] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.960888 14198 consensus_queue.cc:260] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.961148 14198 ts_tablet_manager.cc:1398] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.961313 14198 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:25.962005 14198 tablet_bootstrap.cc:654] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.962020 14201 consensus_queue.cc:260] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:25.962347 14201 ts_tablet_manager.cc:1398] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:25.963184 14199 tablet_bootstrap.cc:492] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:25.963276 14199 ts_tablet_manager.cc:1367] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:25.963327 14197 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:25.963405 14197 ts_tablet_manager.cc:1367] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:25.963696 14199 raft_consensus.cc:348] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:25.963789 14199 raft_consensus.cc:374] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.963816 14199 raft_consensus.cc:729] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.963841 14197 raft_consensus.cc:348] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.963932 14197 raft_consensus.cc:374] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.963958 14197 raft_consensus.cc:729] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.963930 14199 consensus_queue.cc:260] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:25.964177 14199 ts_tablet_manager.cc:1398] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.964187 14197 consensus_queue.cc:260] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.964342 14199 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:25.964524 14197 ts_tablet_manager.cc:1398] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.964670 14197 tablet_bootstrap.cc:492] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:25.965262 14199 tablet_bootstrap.cc:654] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.965363 14197 tablet_bootstrap.cc:654] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:25.966086 14199 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:25.966161 14199 ts_tablet_manager.cc:1367] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.966360 14197 tablet_bootstrap.cc:492] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:25.966454 14197 ts_tablet_manager.cc:1367] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:25.966601 14199 raft_consensus.cc:348] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.966758 14199 raft_consensus.cc:374] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.966836 14199 raft_consensus.cc:729] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.966919 14197 raft_consensus.cc:348] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:25.967038 14197 raft_consensus.cc:374] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.966995 14199 consensus_queue.cc:260] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.967095 14197 raft_consensus.cc:729] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.967231 14197 consensus_queue.cc:260] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:25.967334 14199 ts_tablet_manager.cc:1398] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:25.967613 14197 ts_tablet_manager.cc:1398] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:25.968343 14198 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:25.968423 14198 ts_tablet_manager.cc:1367] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:25.968744 14198 raft_consensus.cc:348] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.968830 14198 raft_consensus.cc:374] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:25.968864 14198 raft_consensus.cc:729] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:25.969038 14198 consensus_queue.cc:260] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:25.969566 14198 ts_tablet_manager.cc:1398] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.021991 13893 raft_consensus.cc:482] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.022109 13893 raft_consensus.cc:504] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.022570 13893 leader_election.cc:288] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:26.022966 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8147fae0de98429ab6554c7e7f5bf8ee" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:26.023010 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8147fae0de98429ab6554c7e7f5bf8ee" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:26.023106 12848 raft_consensus.cc:2454] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:26.023150 12718 raft_consensus.cc:2454] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:26.023377 12916 leader_election.cc:302] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.023598 13893 raft_consensus.cc:2781] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.023658 13893 raft_consensus.cc:482] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.023684 13893 raft_consensus.cc:3037] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.024458 13893 raft_consensus.cc:504] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.024722 13893 leader_election.cc:288] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:26.025014 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8147fae0de98429ab6554c7e7f5bf8ee" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:26.025063 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8147fae0de98429ab6554c7e7f5bf8ee" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:26.025118 12848 raft_consensus.cc:3037] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.025152 12718 raft_consensus.cc:3037] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.025831 12848 raft_consensus.cc:2454] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:26.025943 12718 raft_consensus.cc:2454] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:26.026119 12916 leader_election.cc:302] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.026295 13893 raft_consensus.cc:2781] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.026479 13893 raft_consensus.cc:686] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:26.026607 13893 consensus_queue.cc:237] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.028219 12441 catalog_manager.cc:5484] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:26.076471 13695 raft_consensus.cc:482] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.076579 13695 raft_consensus.cc:504] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.076929 13695 leader_election.cc:288] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:26.077324 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:26.077317 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:26.077457 12848 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:26.077457 12979 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:26.077731 12654 leader_election.cc:302] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.077905 13695 raft_consensus.cc:2781] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.077972 13695 raft_consensus.cc:482] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.078001 13695 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.078893 13695 raft_consensus.cc:504] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.079185 13695 leader_election.cc:288] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:26.079475 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:26.079500 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:26.079571 12848 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.079610 12979 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.080304 12979 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:26.080317 12848 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:26.080607 12654 leader_election.cc:302] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.080999 14212 raft_consensus.cc:2781] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.081089 14212 raft_consensus.cc:686] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:26.081257 14212 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.083154 12441 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:26.166388 14212 raft_consensus.cc:482] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.166507 14212 raft_consensus.cc:504] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.166868 14212 leader_election.cc:288] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:26.167217 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48da29c058b4db3a5b45b25755be1cf" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:26.167284 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48da29c058b4db3a5b45b25755be1cf" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:26.167361 12586 raft_consensus.cc:2454] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:26.167445 12848 raft_consensus.cc:2454] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:26.167667 12653 leader_election.cc:302] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.167902 14212 raft_consensus.cc:2781] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.167974 14212 raft_consensus.cc:482] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.168063 14212 raft_consensus.cc:3037] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.168943 14212 raft_consensus.cc:504] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.169304 14212 leader_election.cc:288] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:26.169550 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48da29c058b4db3a5b45b25755be1cf" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:26.169668 12586 raft_consensus.cc:3037] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.169924 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48da29c058b4db3a5b45b25755be1cf" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:26.170009 12848 raft_consensus.cc:3037] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.170619 12586 raft_consensus.cc:2454] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:26.170912 12653 leader_election.cc:302] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.170953 12848 raft_consensus.cc:2454] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:26.171067 14212 raft_consensus.cc:2781] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.171226 14212 raft_consensus.cc:686] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:26.171376 14212 consensus_queue.cc:237] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.173067 12441 catalog_manager.cc:5484] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:26.297152 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:26.297379 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:26.300777 14215 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:09:26.389870 12718 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.390280 12848 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.390923 12586 raft_consensus.cc:1261] T d48da29c058b4db3a5b45b25755be1cf P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.391000 12979 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.391451 13695 consensus_queue.cc:1035] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.391701 13695 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.391909 12848 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.392302 14210 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.392609 14210 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.391909 12849 raft_consensus.cc:1261] T d48da29c058b4db3a5b45b25755be1cf P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.393478 13695 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.393791 13695 consensus_queue.cc:1035] T d48da29c058b4db3a5b45b25755be1cf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.582 ms
time per row: 0.0587908 ms
I20240501 14:09:26.452899 14250 mvcc.cc:205] Tried to move back new op lower bound from 7022889231926312960 to 7022889230446174208. Current Snapshot: MvccSnapshot[applied={T|T < 7022889231926312960}]
I20240501 14:09:26.460690 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:26.466725 12567 tablet_service.cc:1467] Processing CreateTablet for tablet afa5ebe09c99435fb69ceb527ab4734a (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.466977 12696 tablet_service.cc:1467] Processing CreateTablet for tablet afa5ebe09c99435fb69ceb527ab4734a (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.467099 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5ebe09c99435fb69ceb527ab4734a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.467231 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5ebe09c99435fb69ceb527ab4734a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.469599 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 25552a578587473fa9f22ba47b425bc1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.469887 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25552a578587473fa9f22ba47b425bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.470134 14256 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:26.470955 14256 tablet_bootstrap.cc:654] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.471536 14255 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:26.471736 14258 tablet_bootstrap.cc:492] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:26.471998 14256 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:26.472086 14256 ts_tablet_manager.cc:1367] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.472383 12960 tablet_service.cc:1467] Processing CreateTablet for tablet afa5ebe09c99435fb69ceb527ab4734a (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.472576 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 25552a578587473fa9f22ba47b425bc1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.472637 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5ebe09c99435fb69ceb527ab4734a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.472621 14256 raft_consensus.cc:348] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.472736 14256 raft_consensus.cc:374] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.472761 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25552a578587473fa9f22ba47b425bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.472774 14256 raft_consensus.cc:729] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.473366 14256 consensus_queue.cc:260] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.473687 14258 tablet_bootstrap.cc:654] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.474340 14255 tablet_bootstrap.cc:654] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.473706 14256 ts_tablet_manager.cc:1398] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.474812 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 25552a578587473fa9f22ba47b425bc1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.475057 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25552a578587473fa9f22ba47b425bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.476620 14258 tablet_bootstrap.cc:492] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:26.476706 14258 ts_tablet_manager.cc:1367] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:26.477226 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 16daa4d3a11e47a9ae9764e70a93fa0b (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.477330 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 16daa4d3a11e47a9ae9764e70a93fa0b (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.477471 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16daa4d3a11e47a9ae9764e70a93fa0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.477514 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16daa4d3a11e47a9ae9764e70a93fa0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.477861 14258 raft_consensus.cc:348] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.477978 14258 raft_consensus.cc:374] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.478009 14258 raft_consensus.cc:729] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.478140 14258 consensus_queue.cc:260] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.479029 14258 ts_tablet_manager.cc:1398] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:26.480026 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 16daa4d3a11e47a9ae9764e70a93fa0b (DEFAULT_TABLE table=rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.480244 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16daa4d3a11e47a9ae9764e70a93fa0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.482024 14256 tablet_bootstrap.cc:492] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:26.482156 14260 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:26.482872 14256 tablet_bootstrap.cc:654] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.482985 14260 tablet_bootstrap.cc:654] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.483788 14256 tablet_bootstrap.cc:492] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:26.483867 14256 ts_tablet_manager.cc:1367] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:26.483945 14255 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:26.484038 14255 ts_tablet_manager.cc:1367] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 14:09:26.484272 14256 raft_consensus.cc:348] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.484406 14256 raft_consensus.cc:374] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.484477 14256 raft_consensus.cc:729] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.484464 14255 raft_consensus.cc:348] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.484609 14255 raft_consensus.cc:374] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.484668 14255 raft_consensus.cc:729] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.484661 14256 consensus_queue.cc:260] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.484804 14255 consensus_queue.cc:260] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.485143 14255 ts_tablet_manager.cc:1398] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.485389 14255 tablet_bootstrap.cc:492] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:26.485637 14260 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:26.485755 14260 ts_tablet_manager.cc:1367] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:26.486188 14260 raft_consensus.cc:348] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.486308 14255 tablet_bootstrap.cc:654] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.486384 14260 raft_consensus.cc:374] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.486469 14260 raft_consensus.cc:729] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.486670 14260 consensus_queue.cc:260] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.487061 14260 ts_tablet_manager.cc:1398] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.487247 14260 tablet_bootstrap.cc:492] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:26.487370 14255 tablet_bootstrap.cc:492] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:26.487489 14255 ts_tablet_manager.cc:1367] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:26.487891 14255 raft_consensus.cc:348] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.488085 14260 tablet_bootstrap.cc:654] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.488025 14255 raft_consensus.cc:374] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.488189 14255 raft_consensus.cc:729] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.488359 14255 consensus_queue.cc:260] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.488667 14255 ts_tablet_manager.cc:1398] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.488869 14255 tablet_bootstrap.cc:492] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:26.489542 14260 tablet_bootstrap.cc:492] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:26.489624 14260 ts_tablet_manager.cc:1367] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.489634 14255 tablet_bootstrap.cc:654] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.490283 14260 raft_consensus.cc:348] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.490396 14260 raft_consensus.cc:374] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.490427 14260 raft_consensus.cc:729] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.490535 14260 consensus_queue.cc:260] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.490864 14260 ts_tablet_manager.cc:1398] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:26.491008 14260 tablet_bootstrap.cc:492] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:26.491726 14260 tablet_bootstrap.cc:654] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.492729 14256 ts_tablet_manager.cc:1398] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:26.493391 14255 tablet_bootstrap.cc:492] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:26.493530 14255 ts_tablet_manager.cc:1367] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:26.493626 14260 tablet_bootstrap.cc:492] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:26.493700 14260 ts_tablet_manager.cc:1367] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:26.494028 14255 raft_consensus.cc:348] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.494115 14255 raft_consensus.cc:374] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.494145 14255 raft_consensus.cc:729] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.494102 14260 raft_consensus.cc:348] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.494205 14260 raft_consensus.cc:374] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.494235 14260 raft_consensus.cc:729] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.494271 14255 consensus_queue.cc:260] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.494349 14260 consensus_queue.cc:260] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.494586 14260 ts_tablet_manager.cc:1398] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.495187 14255 ts_tablet_manager.cc:1398] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.517053 14252 raft_consensus.cc:482] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.517195 14252 raft_consensus.cc:504] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.517645 14252 leader_election.cc:288] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:26.518226 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16daa4d3a11e47a9ae9764e70a93fa0b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:26.518229 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16daa4d3a11e47a9ae9764e70a93fa0b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:26.518383 12979 raft_consensus.cc:2454] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:26.518383 12586 raft_consensus.cc:2454] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:26.518687 12653 leader_election.cc:302] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.518914 14252 raft_consensus.cc:2781] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.518981 14252 raft_consensus.cc:482] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.519032 14252 raft_consensus.cc:3037] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.519887 14252 raft_consensus.cc:504] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.520160 14252 leader_election.cc:288] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:26.520480 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16daa4d3a11e47a9ae9764e70a93fa0b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:26.520522 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16daa4d3a11e47a9ae9764e70a93fa0b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:26.520576 12979 raft_consensus.cc:3037] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.520628 12586 raft_consensus.cc:3037] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.521353 12979 raft_consensus.cc:2454] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:26.521461 12586 raft_consensus.cc:2454] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:26.521636 12653 leader_election.cc:302] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.521801 14252 raft_consensus.cc:2781] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.521890 14252 raft_consensus.cc:686] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:26.522022 14252 consensus_queue.cc:237] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.523794 12444 catalog_manager.cc:5484] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:26.525938 14164 raft_consensus.cc:482] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.526038 14164 raft_consensus.cc:504] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.526402 14164 leader_election.cc:288] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:26.526784 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5ebe09c99435fb69ceb527ab4734a" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:26.526779 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5ebe09c99435fb69ceb527ab4734a" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:26.526913 12979 raft_consensus.cc:2454] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:26.526916 12718 raft_consensus.cc:2454] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:26.527156 12520 leader_election.cc:302] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.527371 14164 raft_consensus.cc:2781] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.527434 14164 raft_consensus.cc:482] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.527485 14164 raft_consensus.cc:3037] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.528234 14164 raft_consensus.cc:504] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.528522 14164 leader_election.cc:288] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:26.528839 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5ebe09c99435fb69ceb527ab4734a" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:26.528930 12718 raft_consensus.cc:3037] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.529584 12718 raft_consensus.cc:2454] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:26.529786 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5ebe09c99435fb69ceb527ab4734a" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:26.529865 12520 leader_election.cc:302] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.529899 12979 raft_consensus.cc:3037] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.530017 14164 raft_consensus.cc:2781] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.530128 14164 raft_consensus.cc:686] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:26.530259 14164 consensus_queue.cc:237] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.530910 12979 raft_consensus.cc:2454] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:26.532033 12444 catalog_manager.cc:5484] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:26.543251 14210 raft_consensus.cc:482] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.543362 14210 raft_consensus.cc:504] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.543840 14210 leader_election.cc:288] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:26.544104 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25552a578587473fa9f22ba47b425bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:26.544242 12586 raft_consensus.cc:2454] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:26.544271 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25552a578587473fa9f22ba47b425bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:26.544400 12847 raft_consensus.cc:2454] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:26.544560 12915 leader_election.cc:302] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.544790 14210 raft_consensus.cc:2781] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.544865 14210 raft_consensus.cc:482] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.544914 14210 raft_consensus.cc:3037] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.545884 14210 raft_consensus.cc:504] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.546221 14210 leader_election.cc:288] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:26.546562 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25552a578587473fa9f22ba47b425bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:26.546574 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25552a578587473fa9f22ba47b425bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:26.546677 12586 raft_consensus.cc:3037] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.546686 12847 raft_consensus.cc:3037] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.547426 12586 raft_consensus.cc:2454] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:26.547452 12847 raft_consensus.cc:2454] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:26.547706 12916 leader_election.cc:302] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.547915 14210 raft_consensus.cc:2781] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.547987 14210 raft_consensus.cc:686] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:26.548116 14210 consensus_queue.cc:237] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.549769 12444 catalog_manager.cc:5484] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:26.677574 14272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:26.677801 14272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:26.680819 14272 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 14:09:26.765929 12586 raft_consensus.cc:1261] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.765929 12847 raft_consensus.cc:1261] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.765980 12716 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.766480 12586 raft_consensus.cc:1261] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.766676 12979 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.766717 14164 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.767006 14252 consensus_queue.cc:1035] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.767041 14210 consensus_queue.cc:1035] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.767094 14164 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.767268 14210 consensus_queue.cc:1035] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.771350 12978 raft_consensus.cc:1261] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:26.771664 14212 consensus_queue.cc:1035] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:26.781459 14176 mvcc.cc:205] Tried to move back new op lower bound from 7022889233469161472 to 7022889232582082560. Current Snapshot: MvccSnapshot[applied={T|T < 7022889233469161472}]
I20240501 14:09:26.792189 14161 mvcc.cc:205] Tried to move back new op lower bound from 7022889233466540032 to 7022889232475406336. Current Snapshot: MvccSnapshot[applied={T|T < 7022889233466540032}]
INSERT report
rows total: 2000
time total: 116.88 ms
time per row: 0.0584402 ms
I20240501 14:09:26.844765 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:26.850092 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.850242 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.850406 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.850493 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.850991 12698 tablet_service.cc:1467] Processing CreateTablet for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.851254 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5489eab5b04751bc8b3e5fb9e8cf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.851732 12827 tablet_service.cc:1467] Processing CreateTablet for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.851967 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5489eab5b04751bc8b3e5fb9e8cf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.852636 14258 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:26.853498 14258 tablet_bootstrap.cc:654] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.854451 12960 tablet_service.cc:1467] Processing CreateTablet for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.854691 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5489eab5b04751bc8b3e5fb9e8cf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.855000 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 49f2b0e67d3c43d584ac7583b98caeb9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.855195 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f2b0e67d3c43d584ac7583b98caeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.855124 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 49f2b0e67d3c43d584ac7583b98caeb9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.855394 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f2b0e67d3c43d584ac7583b98caeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.856797 14260 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:26.857015 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.857199 14255 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:26.857235 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.857777 14260 tablet_bootstrap.cc:654] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.858016 14258 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:26.858085 14255 tablet_bootstrap.cc:654] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.858093 14258 ts_tablet_manager.cc:1367] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:26.858628 14258 raft_consensus.cc:348] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.858772 14258 raft_consensus.cc:374] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.858816 14258 raft_consensus.cc:729] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.858938 14258 consensus_queue.cc:260] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.859105 14260 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:26.859181 14260 ts_tablet_manager.cc:1367] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:26.859205 14255 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:26.859277 14255 ts_tablet_manager.cc:1367] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.859282 14258 ts_tablet_manager.cc:1398] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.859449 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 49f2b0e67d3c43d584ac7583b98caeb9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:26.859658 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f2b0e67d3c43d584ac7583b98caeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:26.859679 14255 raft_consensus.cc:348] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.859799 14255 raft_consensus.cc:374] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.859834 14255 raft_consensus.cc:729] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.859947 14255 consensus_queue.cc:260] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.860214 14255 ts_tablet_manager.cc:1398] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.860427 14255 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:26.860541 14258 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:26.861313 14255 tablet_bootstrap.cc:654] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.861397 14258 tablet_bootstrap.cc:654] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.861511 14256 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:26.859589 14260 raft_consensus.cc:348] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.862315 14260 raft_consensus.cc:374] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.862347 14256 tablet_bootstrap.cc:654] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.862362 14260 raft_consensus.cc:729] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.862480 14260 consensus_queue.cc:260] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.862603 14258 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:26.862681 14258 ts_tablet_manager.cc:1367] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:26.862824 14260 ts_tablet_manager.cc:1398] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:26.863008 14260 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:26.863193 14258 raft_consensus.cc:348] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.863289 14258 raft_consensus.cc:374] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.863322 14258 raft_consensus.cc:729] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.863422 14258 consensus_queue.cc:260] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.863518 14256 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:26.863584 14256 ts_tablet_manager.cc:1367] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.863687 14258 ts_tablet_manager.cc:1398] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.863859 14258 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:26.863886 14260 tablet_bootstrap.cc:654] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.864004 14256 raft_consensus.cc:348] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.864099 14256 raft_consensus.cc:374] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.864130 14256 raft_consensus.cc:729] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.864238 14256 consensus_queue.cc:260] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.864542 14256 ts_tablet_manager.cc:1398] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.864696 14256 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:26.864874 14258 tablet_bootstrap.cc:654] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.865621 14256 tablet_bootstrap.cc:654] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:26.869089 14255 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:26.869181 14255 ts_tablet_manager.cc:1367] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:26.869594 14255 raft_consensus.cc:348] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.869709 14255 raft_consensus.cc:374] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.869750 14255 raft_consensus.cc:729] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.869871 14255 consensus_queue.cc:260] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.870152 14255 ts_tablet_manager.cc:1398] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.871135 14256 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:26.871210 14256 ts_tablet_manager.cc:1367] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:26.871735 14256 raft_consensus.cc:348] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.871845 14256 raft_consensus.cc:374] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.871881 14256 raft_consensus.cc:729] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.871994 14256 consensus_queue.cc:260] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.872277 14260 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:26.872277 14256 ts_tablet_manager.cc:1398] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:26.872356 14260 ts_tablet_manager.cc:1367] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:26.872841 14260 raft_consensus.cc:348] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.872943 14260 raft_consensus.cc:374] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.872974 14260 raft_consensus.cc:729] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.873090 14260 consensus_queue.cc:260] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.874073 14260 ts_tablet_manager.cc:1398] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:26.874868 14258 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:26.874967 14258 ts_tablet_manager.cc:1367] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:26.875447 14258 raft_consensus.cc:348] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.875571 14258 raft_consensus.cc:374] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:26.875619 14258 raft_consensus.cc:729] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:26.875761 14258 consensus_queue.cc:260] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.876063 14258 ts_tablet_manager.cc:1398] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:26.914572 14310 raft_consensus.cc:482] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.914691 14310 raft_consensus.cc:504] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.915102 14310 leader_election.cc:288] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:26.915524 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f2b0e67d3c43d584ac7583b98caeb9" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:26.915525 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f2b0e67d3c43d584ac7583b98caeb9" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:26.915670 12847 raft_consensus.cc:2454] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:26.915669 12587 raft_consensus.cc:2454] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:26.915977 12916 leader_election.cc:302] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.916146 14310 raft_consensus.cc:2781] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.916208 14310 raft_consensus.cc:482] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.916234 14310 raft_consensus.cc:3037] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.917088 14310 raft_consensus.cc:504] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.917392 14310 leader_election.cc:288] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:26.917685 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f2b0e67d3c43d584ac7583b98caeb9" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:26.917685 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f2b0e67d3c43d584ac7583b98caeb9" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:26.917809 12587 raft_consensus.cc:3037] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.917805 12847 raft_consensus.cc:3037] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.918756 12847 raft_consensus.cc:2454] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:26.918798 12587 raft_consensus.cc:2454] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:26.919049 12916 leader_election.cc:302] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.919206 14310 raft_consensus.cc:2781] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.919312 14310 raft_consensus.cc:686] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:26.919452 14310 consensus_queue.cc:237] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:26.921247 12445 catalog_manager.cc:5484] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:26.930509 14264 raft_consensus.cc:482] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.930663 14264 raft_consensus.cc:504] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.931106 14264 leader_election.cc:288] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:26.931493 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5489eab5b04751bc8b3e5fb9e8cf73" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:26.931500 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5489eab5b04751bc8b3e5fb9e8cf73" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:26.931654 12978 raft_consensus.cc:2454] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:26.931650 12718 raft_consensus.cc:2454] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:26.931989 12784 leader_election.cc:302] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:26.932210 14264 raft_consensus.cc:2781] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.932272 14264 raft_consensus.cc:482] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.932329 14264 raft_consensus.cc:3037] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.933415 14264 raft_consensus.cc:504] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.933761 14264 leader_election.cc:288] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:26.934175 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5489eab5b04751bc8b3e5fb9e8cf73" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:26.934141 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5489eab5b04751bc8b3e5fb9e8cf73" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:26.934312 12978 raft_consensus.cc:3037] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.934312 12718 raft_consensus.cc:3037] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.935297 12978 raft_consensus.cc:2454] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:26.935297 12718 raft_consensus.cc:2454] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:26.935606 12782 leader_election.cc:302] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.935846 14264 raft_consensus.cc:2781] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.935930 14264 raft_consensus.cc:686] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:26.936035 14264 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:26.938043 12445 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:26.972517 14264 raft_consensus.cc:482] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:26.972635 14264 raft_consensus.cc:504] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.973001 14264 leader_election.cc:288] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:26.973400 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:26.973428 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:26.973570 12587 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:26.973572 12718 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:26.973886 12782 leader_election.cc:302] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:26.974074 14264 raft_consensus.cc:2781] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:26.974121 14264 raft_consensus.cc:482] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:26.974162 14264 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.974977 14264 raft_consensus.cc:504] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.975265 14264 leader_election.cc:288] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:26.975579 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:26.975579 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:26.975678 12587 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.975687 12718 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:26.976476 12587 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:26.976601 12718 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:26.976801 12784 leader_election.cc:302] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:26.976950 14264 raft_consensus.cc:2781] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:26.977042 14264 raft_consensus.cc:686] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:26.977187 14264 consensus_queue.cc:237] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:26.979103 12445 catalog_manager.cc:5484] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:27.092025 14325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:27.092249 14325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:27.095237 14325 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 14:09:27.175915 12979 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.176327 14264 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.179777 12587 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.180291 12847 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.180382 12718 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.180819 13893 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.181265 14196 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.183409 12587 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.183948 13893 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.184756 12716 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.185714 14264 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.185963 14264 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.197005 14172 mvcc.cc:205] Tried to move back new op lower bound from 7022889235148283904 to 7022889234171432960. Current Snapshot: MvccSnapshot[applied={T|T < 7022889235148283904}]
INSERT report
rows total: 2000
time total: 100.729 ms
time per row: 0.0503644 ms
I20240501 14:09:27.241050 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:27.247627 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.247627 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.247797 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.247876 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.247995 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6305f3f7cb8846bf843d2d753fd1ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.248083 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6305f3f7cb8846bf843d2d753fd1ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.248101 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.249572 14256 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:27.249748 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.250187 12565 tablet_service.cc:1467] Processing CreateTablet for tablet bfc871f734904165ab772321b04a65a4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.250321 12960 tablet_service.cc:1467] Processing CreateTablet for tablet bfc871f734904165ab772321b04a65a4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.250403 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc871f734904165ab772321b04a65a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.250504 14256 tablet_bootstrap.cc:654] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.250519 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc871f734904165ab772321b04a65a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.252260 14255 tablet_bootstrap.cc:492] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:27.252399 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.252584 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6305f3f7cb8846bf843d2d753fd1ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.253047 14255 tablet_bootstrap.cc:654] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.254065 14255 tablet_bootstrap.cc:492] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:27.254123 14255 ts_tablet_manager.cc:1367] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:27.254498 14260 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:27.254501 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.254567 14255 raft_consensus.cc:348] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.254673 14255 raft_consensus.cc:374] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.254709 14255 raft_consensus.cc:729] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.254802 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.254858 14255 consensus_queue.cc:260] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.255151 14255 ts_tablet_manager.cc:1398] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:27.255312 14255 tablet_bootstrap.cc:492] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:27.255312 14260 tablet_bootstrap.cc:654] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.256400 14255 tablet_bootstrap.cc:654] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.256366 12827 tablet_service.cc:1467] Processing CreateTablet for tablet bfc871f734904165ab772321b04a65a4 (DEFAULT_TABLE table=rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.256616 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfc871f734904165ab772321b04a65a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.257056 14258 tablet_bootstrap.cc:492] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:27.257688 14255 tablet_bootstrap.cc:492] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:27.257834 14255 ts_tablet_manager.cc:1367] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:27.258186 14258 tablet_bootstrap.cc:654] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.259313 14255 raft_consensus.cc:348] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.259572 14255 raft_consensus.cc:374] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.259702 14255 raft_consensus.cc:729] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.260020 14255 consensus_queue.cc:260] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.260540 14255 ts_tablet_manager.cc:1398] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:27.260826 14255 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:27.262171 14260 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:27.262274 14260 ts_tablet_manager.cc:1367] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:27.262329 14255 tablet_bootstrap.cc:654] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.262843 14260 raft_consensus.cc:348] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.262961 14260 raft_consensus.cc:374] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.263000 14260 raft_consensus.cc:729] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.263113 14260 consensus_queue.cc:260] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.263594 14260 ts_tablet_manager.cc:1398] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:27.263602 14255 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:27.263669 14255 ts_tablet_manager.cc:1367] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:27.263733 14260 tablet_bootstrap.cc:492] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:27.264039 14255 raft_consensus.cc:348] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.264117 14255 raft_consensus.cc:374] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.264142 14255 raft_consensus.cc:729] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.264235 14255 consensus_queue.cc:260] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.264441 14255 ts_tablet_manager.cc:1398] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:27.264514 14260 tablet_bootstrap.cc:654] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.265549 14260 tablet_bootstrap.cc:492] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:27.265616 14260 ts_tablet_manager.cc:1367] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:27.266008 14260 raft_consensus.cc:348] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.266086 14260 raft_consensus.cc:374] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.266116 14260 raft_consensus.cc:729] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.266248 14260 consensus_queue.cc:260] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.266548 14260 ts_tablet_manager.cc:1398] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.267715 14258 tablet_bootstrap.cc:492] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:27.267810 14258 ts_tablet_manager.cc:1367] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:09:27.268240 14258 raft_consensus.cc:348] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.268306 14258 raft_consensus.cc:374] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.268324 14258 raft_consensus.cc:729] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.268435 14258 consensus_queue.cc:260] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.268682 14258 ts_tablet_manager.cc:1398] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.268837 14258 tablet_bootstrap.cc:492] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:27.269547 14258 tablet_bootstrap.cc:654] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.270666 14258 tablet_bootstrap.cc:492] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:27.270771 14258 ts_tablet_manager.cc:1367] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:27.271220 14258 raft_consensus.cc:348] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.271328 14258 raft_consensus.cc:374] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.271358 14258 raft_consensus.cc:729] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.271471 14256 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:27.271564 14256 ts_tablet_manager.cc:1367] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:27.271670 14258 consensus_queue.cc:260] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.271927 14258 ts_tablet_manager.cc:1398] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.272042 14256 raft_consensus.cc:348] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.272123 14256 raft_consensus.cc:374] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.272142 14256 raft_consensus.cc:729] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.272245 14256 consensus_queue.cc:260] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.272506 14256 ts_tablet_manager.cc:1398] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:27.273236 14256 tablet_bootstrap.cc:492] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:27.274143 14256 tablet_bootstrap.cc:654] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.275214 14256 tablet_bootstrap.cc:492] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:27.275292 14256 ts_tablet_manager.cc:1367] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:27.275693 14256 raft_consensus.cc:348] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.275794 14256 raft_consensus.cc:374] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.275826 14256 raft_consensus.cc:729] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.275929 14256 consensus_queue.cc:260] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.276180 14256 ts_tablet_manager.cc:1398] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.349870 14252 raft_consensus.cc:482] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.350008 14252 raft_consensus.cc:504] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.350340 14252 leader_election.cc:288] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.350756 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6305f3f7cb8846bf843d2d753fd1ae07" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:27.350776 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6305f3f7cb8846bf843d2d753fd1ae07" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:27.350894 12978 raft_consensus.cc:2454] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:27.350924 12586 raft_consensus.cc:2454] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:27.351182 12653 leader_election.cc:302] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:27.351362 14252 raft_consensus.cc:2781] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.351426 14252 raft_consensus.cc:482] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.351476 14252 raft_consensus.cc:3037] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.352267 14252 raft_consensus.cc:504] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.352625 14252 leader_election.cc:288] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.352967 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6305f3f7cb8846bf843d2d753fd1ae07" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:27.352969 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6305f3f7cb8846bf843d2d753fd1ae07" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:27.353087 12978 raft_consensus.cc:3037] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.353087 12586 raft_consensus.cc:3037] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.353996 12586 raft_consensus.cc:2454] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:27.353994 12978 raft_consensus.cc:2454] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:27.354300 12653 leader_election.cc:302] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:27.354454 14212 raft_consensus.cc:2781] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.354547 14212 raft_consensus.cc:686] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:27.354699 14212 consensus_queue.cc:237] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:27.356626 12441 catalog_manager.cc:5484] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:27.393378 14359 raft_consensus.cc:482] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.393509 14359 raft_consensus.cc:504] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.393986 14359 leader_election.cc:288] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.394414 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc871f734904165ab772321b04a65a4" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:27.394481 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc871f734904165ab772321b04a65a4" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:27.394593 12978 raft_consensus.cc:2454] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:27.394644 12586 raft_consensus.cc:2454] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:27.394942 12784 leader_election.cc:302] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:27.395124 14359 raft_consensus.cc:2781] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.395195 14359 raft_consensus.cc:482] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.395227 14359 raft_consensus.cc:3037] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.396289 14359 raft_consensus.cc:504] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.396610 14359 leader_election.cc:288] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.396919 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc871f734904165ab772321b04a65a4" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:27.396905 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfc871f734904165ab772321b04a65a4" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:27.397043 12978 raft_consensus.cc:3037] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.397063 12587 raft_consensus.cc:3037] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.397919 12587 raft_consensus.cc:2454] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:27.397920 12978 raft_consensus.cc:2454] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:27.398209 12784 leader_election.cc:302] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:27.398388 14359 raft_consensus.cc:2781] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.398488 14359 raft_consensus.cc:686] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:27.398636 14359 consensus_queue.cc:237] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.400950 12441 catalog_manager.cc:5484] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:27.516019 14359 raft_consensus.cc:482] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.516165 14359 raft_consensus.cc:504] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.516552 14359 leader_election.cc:288] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.516918 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74c6b1b2dc4793a8d7ed4c41de013d" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:27.516952 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74c6b1b2dc4793a8d7ed4c41de013d" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:27.517079 12586 raft_consensus.cc:2454] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:27.517074 12716 raft_consensus.cc:2454] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:27.517340 12784 leader_election.cc:302] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:27.517506 14359 raft_consensus.cc:2781] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.517565 14359 raft_consensus.cc:482] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.517609 14359 raft_consensus.cc:3037] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.518381 14359 raft_consensus.cc:504] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.518708 14359 leader_election.cc:288] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.519039 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74c6b1b2dc4793a8d7ed4c41de013d" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:27.519044 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74c6b1b2dc4793a8d7ed4c41de013d" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:27.519136 12716 raft_consensus.cc:3037] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.519143 12586 raft_consensus.cc:3037] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.519840 12716 raft_consensus.cc:2454] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:27.519893 12586 raft_consensus.cc:2454] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:27.520085 12782 leader_election.cc:302] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:27.520303 14359 raft_consensus.cc:2781] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.520376 14359 raft_consensus.cc:686] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:27.520529 14359 consensus_queue.cc:237] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.522394 12441 catalog_manager.cc:5484] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:27.659953 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:27.660199 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:27.663453 14378 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 14:09:27.757979 12587 raft_consensus.cc:1261] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.758106 12716 raft_consensus.cc:1261] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.758369 14359 consensus_queue.cc:1035] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.758560 14360 consensus_queue.cc:1035] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.760962 12586 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.761431 14252 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.761565 12978 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.762977 14252 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.774259 12585 raft_consensus.cc:1261] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.776078 12979 raft_consensus.cc:1261] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:27.776438 14359 consensus_queue.cc:1035] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:27.777737 14359 consensus_queue.cc:1035] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 127.321 ms
time per row: 0.0636605 ms
I20240501 14:09:27.830860 12444 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:27.836405 12698 tablet_service.cc:1467] Processing CreateTablet for tablet abac52c109174021ac11360b0d22ccfa (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.836737 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abac52c109174021ac11360b0d22ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.837690 12960 tablet_service.cc:1467] Processing CreateTablet for tablet abac52c109174021ac11360b0d22ccfa (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.837999 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abac52c109174021ac11360b0d22ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.838940 14438 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:27.839385 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 40b29fb993f34e99ad3747a2ec7d6c4f (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.839627 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b29fb993f34e99ad3747a2ec7d6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.839635 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 6628290de33e4aba8ccd1651d1d054dd (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.839804 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 40b29fb993f34e99ad3747a2ec7d6c4f (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.839928 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628290de33e4aba8ccd1651d1d054dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.839635 12827 tablet_service.cc:1467] Processing CreateTablet for tablet abac52c109174021ac11360b0d22ccfa (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.840189 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abac52c109174021ac11360b0d22ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.840245 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 40b29fb993f34e99ad3747a2ec7d6c4f (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.840521 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b29fb993f34e99ad3747a2ec7d6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.842119 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b29fb993f34e99ad3747a2ec7d6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.842947 14438 tablet_bootstrap.cc:654] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.842907 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 6628290de33e4aba8ccd1651d1d054dd (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.843171 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628290de33e4aba8ccd1651d1d054dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.843384 14441 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:27.844108 14438 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:27.844189 14438 ts_tablet_manager.cc:1367] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:27.844570 14441 tablet_bootstrap.cc:654] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.844578 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 6628290de33e4aba8ccd1651d1d054dd (DEFAULT_TABLE table=rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:27.844574 14438 raft_consensus.cc:348] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:27.844717 14438 raft_consensus.cc:374] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.844745 14438 raft_consensus.cc:729] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.844827 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628290de33e4aba8ccd1651d1d054dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:27.844873 14438 consensus_queue.cc:260] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:27.845137 14438 ts_tablet_manager.cc:1398] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.845288 14438 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:27.845901 14439 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:27.846036 14438 tablet_bootstrap.cc:654] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.846797 14439 tablet_bootstrap.cc:654] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.847024 14440 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:27.847854 14439 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:27.847937 14439 ts_tablet_manager.cc:1367] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:27.847944 14440 tablet_bootstrap.cc:654] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.848019 14438 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:27.848090 14438 ts_tablet_manager.cc:1367] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:27.848317 14439 raft_consensus.cc:348] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:27.848397 14439 raft_consensus.cc:374] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.848415 14439 raft_consensus.cc:729] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.848516 14439 consensus_queue.cc:260] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:27.848542 14438 raft_consensus.cc:348] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.848647 14438 raft_consensus.cc:374] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.848677 14438 raft_consensus.cc:729] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.848778 14439 ts_tablet_manager.cc:1398] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:27.848819 14438 consensus_queue.cc:260] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.848932 14439 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:27.849174 14438 ts_tablet_manager.cc:1398] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.849318 14438 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:27.849684 14439 tablet_bootstrap.cc:654] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.849995 14441 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:27.850071 14441 ts_tablet_manager.cc:1367] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:27.850095 14438 tablet_bootstrap.cc:654] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.850495 14441 raft_consensus.cc:348] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.850606 14441 raft_consensus.cc:374] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.851313 14438 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:27.851392 14438 ts_tablet_manager.cc:1367] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:27.850639 14441 raft_consensus.cc:729] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.851694 14441 consensus_queue.cc:260] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.851765 14438 raft_consensus.cc:348] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:27.851871 14438 raft_consensus.cc:374] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.851912 14438 raft_consensus.cc:729] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.852025 14439 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:27.852013 14438 consensus_queue.cc:260] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:27.852090 14439 ts_tablet_manager.cc:1367] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:27.852277 14438 ts_tablet_manager.cc:1398] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:27.852543 14439 raft_consensus.cc:348] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:27.852656 14439 raft_consensus.cc:374] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.852690 14439 raft_consensus.cc:729] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.852742 14441 ts_tablet_manager.cc:1398] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:27.852808 14439 consensus_queue.cc:260] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:27.852989 14440 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:27.853080 14440 ts_tablet_manager.cc:1367] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:27.853091 14439 ts_tablet_manager.cc:1398] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:27.853535 14440 raft_consensus.cc:348] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:27.853643 14440 raft_consensus.cc:374] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.853678 14440 raft_consensus.cc:729] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.853797 14440 consensus_queue.cc:260] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:27.854086 14440 ts_tablet_manager.cc:1398] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:27.854220 14440 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:27.855247 14440 tablet_bootstrap.cc:654] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.856243 14440 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:27.856317 14440 ts_tablet_manager.cc:1367] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:27.857015 14440 raft_consensus.cc:348] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:27.857111 14440 raft_consensus.cc:374] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.857151 14440 raft_consensus.cc:729] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.857255 14440 consensus_queue.cc:260] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:27.857548 14440 ts_tablet_manager.cc:1398] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:27.857710 14440 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:27.858561 14440 tablet_bootstrap.cc:654] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:27.859601 14440 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:27.859691 14440 ts_tablet_manager.cc:1367] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:27.860042 14440 raft_consensus.cc:348] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.860129 14440 raft_consensus.cc:374] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:27.860163 14440 raft_consensus.cc:729] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:27.860275 14440 consensus_queue.cc:260] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.860567 14440 ts_tablet_manager.cc:1398] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:27.975816 14212 raft_consensus.cc:482] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.975924 14212 raft_consensus.cc:504] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.976267 14212 leader_election.cc:288] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.976605 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b29fb993f34e99ad3747a2ec7d6c4f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:27.976701 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b29fb993f34e99ad3747a2ec7d6c4f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:27.976740 12849 raft_consensus.cc:2454] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:27.976851 12587 raft_consensus.cc:2454] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:27.977062 12654 leader_election.cc:302] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:27.977216 14212 raft_consensus.cc:2781] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:27.977278 14212 raft_consensus.cc:482] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:27.977306 14212 raft_consensus.cc:3037] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.978107 14212 raft_consensus.cc:504] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.978400 14212 leader_election.cc:288] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:27.978703 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b29fb993f34e99ad3747a2ec7d6c4f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:27.978780 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40b29fb993f34e99ad3747a2ec7d6c4f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:27.978842 12849 raft_consensus.cc:3037] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.978914 12587 raft_consensus.cc:3037] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:27.979854 12587 raft_consensus.cc:2454] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:27.979858 12849 raft_consensus.cc:2454] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:27.980113 12654 leader_election.cc:302] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:27.980238 14212 raft_consensus.cc:2781] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:27.980316 14212 raft_consensus.cc:686] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:27.980463 14212 consensus_queue.cc:237] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:27.982321 12444 catalog_manager.cc:5484] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:28.057703 14212 raft_consensus.cc:482] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.057842 14212 raft_consensus.cc:504] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.058287 14212 leader_election.cc:288] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:28.058710 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abac52c109174021ac11360b0d22ccfa" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:28.058892 12849 raft_consensus.cc:2454] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:28.058974 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abac52c109174021ac11360b0d22ccfa" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:28.059134 12978 raft_consensus.cc:2454] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:28.059249 12654 leader_election.cc:302] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.059441 14212 raft_consensus.cc:2781] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.059499 14212 raft_consensus.cc:482] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.059551 14212 raft_consensus.cc:3037] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.060438 14212 raft_consensus.cc:504] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.060971 14212 leader_election.cc:288] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:28.061026 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abac52c109174021ac11360b0d22ccfa" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:28.061070 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abac52c109174021ac11360b0d22ccfa" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:28.061125 12978 raft_consensus.cc:3037] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.061187 12849 raft_consensus.cc:3037] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.062011 12849 raft_consensus.cc:2454] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:28.062063 12978 raft_consensus.cc:2454] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:28.062305 12654 leader_election.cc:302] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.062472 14252 raft_consensus.cc:2781] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.062676 14252 raft_consensus.cc:686] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:28.062820 14252 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.064774 12444 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:28.070134 14252 raft_consensus.cc:482] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.070217 14252 raft_consensus.cc:504] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.070525 14252 leader_election.cc:288] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:28.070837 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:28.070887 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:28.070966 12978 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:28.070992 12849 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:28.071280 12653 leader_election.cc:302] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.071442 14252 raft_consensus.cc:2781] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.071499 14252 raft_consensus.cc:482] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.071539 14252 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.072279 14252 raft_consensus.cc:504] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.072614 14252 leader_election.cc:288] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:28.072916 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:28.072916 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:28.073035 12849 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.073032 12978 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.073944 12978 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:28.073945 12849 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:28.074190 12654 leader_election.cc:302] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.074390 14252 raft_consensus.cc:2781] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.074474 14252 raft_consensus.cc:686] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:28.074595 14252 consensus_queue.cc:237] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.076261 12444 catalog_manager.cc:5484] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:28.188704 14455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:28.188952 14455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:28.192009 14455 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240501 14:09:28.271901 12849 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.272034 12847 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.272495 12979 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.272485 14252 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.272729 14452 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.273332 14452 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.275629 12587 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.277344 12848 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.277922 14252 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.280419 12978 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.281172 14212 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.290170 14452 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.51 ms
time per row: 0.0567549 ms
I20240501 14:09:28.348160 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:28.353653 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 82a0a004a4ed4e05b4a099b1002372c6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.353971 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a0a004a4ed4e05b4a099b1002372c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.354598 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 82a0a004a4ed4e05b4a099b1002372c6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.354647 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 82a0a004a4ed4e05b4a099b1002372c6 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.354880 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a0a004a4ed4e05b4a099b1002372c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.354957 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a0a004a4ed4e05b4a099b1002372c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.356164 14494 tablet_bootstrap.cc:492] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:28.356848 14495 tablet_bootstrap.cc:492] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:28.356812 12696 tablet_service.cc:1467] Processing CreateTablet for tablet d2631324f66d454697b895ecdd227276 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.357055 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2631324f66d454697b895ecdd227276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.357241 12828 tablet_service.cc:1467] Processing CreateTablet for tablet a9f2c6558161436683f50486e437dc28 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.357530 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f2c6558161436683f50486e437dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.357537 14496 tablet_bootstrap.cc:492] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:28.357993 14495 tablet_bootstrap.cc:654] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.358373 14496 tablet_bootstrap.cc:654] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.359534 14440 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:28.359520 12959 tablet_service.cc:1467] Processing CreateTablet for tablet a9f2c6558161436683f50486e437dc28 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.359519 14496 tablet_bootstrap.cc:492] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:28.359748 14496 ts_tablet_manager.cc:1367] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:28.359789 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f2c6558161436683f50486e437dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.360172 14496 raft_consensus.cc:348] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.360271 14496 raft_consensus.cc:374] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.360301 14496 raft_consensus.cc:729] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.360467 14496 consensus_queue.cc:260] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.360527 14440 tablet_bootstrap.cc:654] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.360795 14496 ts_tablet_manager.cc:1398] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.360954 14496 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:28.361303 12566 tablet_service.cc:1467] Processing CreateTablet for tablet a9f2c6558161436683f50486e437dc28 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.361542 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f2c6558161436683f50486e437dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.361909 14496 tablet_bootstrap.cc:654] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.361917 12960 tablet_service.cc:1467] Processing CreateTablet for tablet d2631324f66d454697b895ecdd227276 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.362126 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2631324f66d454697b895ecdd227276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.362953 14496 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:28.363030 14496 ts_tablet_manager.cc:1367] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:28.363298 14495 tablet_bootstrap.cc:492] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:28.363358 14495 ts_tablet_manager.cc:1367] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:28.363444 14496 raft_consensus.cc:348] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.363534 14496 raft_consensus.cc:374] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.363569 14496 raft_consensus.cc:729] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.363706 14496 consensus_queue.cc:260] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.363751 14495 raft_consensus.cc:348] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.363830 14495 raft_consensus.cc:374] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.363858 14495 raft_consensus.cc:729] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.363947 14496 ts_tablet_manager.cc:1398] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:28.363955 14495 consensus_queue.cc:260] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.364169 14495 ts_tablet_manager.cc:1398] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:28.364269 14495 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:28.364305 12565 tablet_service.cc:1467] Processing CreateTablet for tablet d2631324f66d454697b895ecdd227276 (DEFAULT_TABLE table=rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.364493 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2631324f66d454697b895ecdd227276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.364863 14495 tablet_bootstrap.cc:654] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.365764 14495 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:28.365847 14495 ts_tablet_manager.cc:1367] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.366367 14495 raft_consensus.cc:348] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.366469 14495 raft_consensus.cc:374] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.366500 14495 raft_consensus.cc:729] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.366608 14495 consensus_queue.cc:260] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.366904 14495 ts_tablet_manager.cc:1398] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.367048 14495 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:28.367837 14495 tablet_bootstrap.cc:654] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.370134 14440 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:28.370223 14440 ts_tablet_manager.cc:1367] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240501 14:09:28.370666 14440 raft_consensus.cc:348] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.370808 14440 raft_consensus.cc:374] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.370847 14440 raft_consensus.cc:729] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.370962 14440 consensus_queue.cc:260] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.371222 14440 ts_tablet_manager.cc:1398] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.372874 14495 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:28.372963 14495 ts_tablet_manager.cc:1367] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:28.373354 14495 raft_consensus.cc:348] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.373495 14495 raft_consensus.cc:374] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.373549 14495 raft_consensus.cc:729] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.373529 14494 tablet_bootstrap.cc:654] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.373675 14495 consensus_queue.cc:260] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.373978 14495 ts_tablet_manager.cc:1398] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.375447 14494 tablet_bootstrap.cc:492] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:28.375536 14494 ts_tablet_manager.cc:1367] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:28.375907 14494 raft_consensus.cc:348] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.376031 14494 raft_consensus.cc:374] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.376077 14494 raft_consensus.cc:729] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.376196 14494 consensus_queue.cc:260] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.377521 14494 ts_tablet_manager.cc:1398] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.377682 14494 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:28.378422 14494 tablet_bootstrap.cc:654] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.379971 14494 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:28.380052 14494 ts_tablet_manager.cc:1367] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:28.380429 14494 raft_consensus.cc:348] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.380527 14494 raft_consensus.cc:374] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.380565 14494 raft_consensus.cc:729] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.380671 14494 consensus_queue.cc:260] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.380995 14494 ts_tablet_manager.cc:1398] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.381501 14494 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:28.382194 14494 tablet_bootstrap.cc:654] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.383139 14494 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:28.383216 14494 ts_tablet_manager.cc:1367] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.383575 14494 raft_consensus.cc:348] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.383654 14494 raft_consensus.cc:374] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.383684 14494 raft_consensus.cc:729] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.383792 14494 consensus_queue.cc:260] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.384137 14494 ts_tablet_manager.cc:1398] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.427060 14210 raft_consensus.cc:482] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.427170 14210 raft_consensus.cc:504] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.427498 14210 leader_election.cc:288] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:28.427827 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2631324f66d454697b895ecdd227276" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:28.427963 12718 raft_consensus.cc:2454] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:28.427928 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2631324f66d454697b895ecdd227276" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:28.428069 12585 raft_consensus.cc:2454] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:28.428285 12913 leader_election.cc:302] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.428439 14210 raft_consensus.cc:2781] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.428503 14210 raft_consensus.cc:482] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.428536 14210 raft_consensus.cc:3037] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.429292 14210 raft_consensus.cc:504] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.429587 14210 leader_election.cc:288] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:28.429905 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2631324f66d454697b895ecdd227276" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:28.429905 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2631324f66d454697b895ecdd227276" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:28.430029 12718 raft_consensus.cc:3037] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.430054 12585 raft_consensus.cc:3037] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.431030 12718 raft_consensus.cc:2454] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:28.431030 12585 raft_consensus.cc:2454] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:28.431299 12915 leader_election.cc:302] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:28.431444 14210 raft_consensus.cc:2781] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.431525 14210 raft_consensus.cc:686] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:28.431654 14210 consensus_queue.cc:237] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:28.433436 12440 catalog_manager.cc:5484] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:28.444521 14360 raft_consensus.cc:482] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.444628 14360 raft_consensus.cc:504] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.444940 14360 leader_election.cc:288] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:28.445276 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f2c6558161436683f50486e437dc28" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:28.445286 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f2c6558161436683f50486e437dc28" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:28.445397 12585 raft_consensus.cc:2454] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:28.445415 12978 raft_consensus.cc:2454] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:28.445672 12784 leader_election.cc:302] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:28.445854 14360 raft_consensus.cc:2781] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.445912 14360 raft_consensus.cc:482] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.445938 14360 raft_consensus.cc:3037] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.446695 14360 raft_consensus.cc:504] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.447006 14360 leader_election.cc:288] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:28.447276 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f2c6558161436683f50486e437dc28" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:28.447300 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f2c6558161436683f50486e437dc28" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:28.447387 12585 raft_consensus.cc:3037] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.447389 12978 raft_consensus.cc:3037] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.448043 12978 raft_consensus.cc:2454] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:28.448056 12585 raft_consensus.cc:2454] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:28.448285 12784 leader_election.cc:302] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:28.448488 14360 raft_consensus.cc:2781] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.448565 14360 raft_consensus.cc:686] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:28.448698 14360 consensus_queue.cc:237] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:28.450507 12443 catalog_manager.cc:5484] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:28.528687 14212 raft_consensus.cc:482] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.528796 14212 raft_consensus.cc:504] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.529145 14212 leader_election.cc:288] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:28.529455 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a0a004a4ed4e05b4a099b1002372c6" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:28.529539 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a0a004a4ed4e05b4a099b1002372c6" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:28.529583 12585 raft_consensus.cc:2454] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:28.529667 12978 raft_consensus.cc:2454] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:28.529870 12653 leader_election.cc:302] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.530037 14212 raft_consensus.cc:2781] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.530092 14212 raft_consensus.cc:482] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.530128 14212 raft_consensus.cc:3037] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.531090 14212 raft_consensus.cc:504] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.531394 14212 leader_election.cc:288] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:28.531682 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a0a004a4ed4e05b4a099b1002372c6" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:28.531723 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a0a004a4ed4e05b4a099b1002372c6" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:28.531816 12978 raft_consensus.cc:3037] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.531811 12585 raft_consensus.cc:3037] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.532739 12978 raft_consensus.cc:2454] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:28.532739 12585 raft_consensus.cc:2454] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:28.533053 12653 leader_election.cc:302] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.533267 14212 raft_consensus.cc:2781] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.533373 14212 raft_consensus.cc:686] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:28.533510 14212 consensus_queue.cc:237] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.535403 12443 catalog_manager.cc:5484] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:28.666093 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:28.666332 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:28.669167 14509 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 14:09:28.744199 12978 raft_consensus.cc:1261] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.744199 12585 raft_consensus.cc:1261] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.744601 14452 consensus_queue.cc:1035] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.744753 14212 consensus_queue.cc:1035] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.749539 12587 raft_consensus.cc:1261] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.749539 12979 raft_consensus.cc:1261] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.750002 14360 consensus_queue.cc:1035] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.750134 14429 consensus_queue.cc:1035] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.767452 12585 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.768116 14486 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.768292 12718 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:28.768838 14486 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:28.804273 14422 mvcc.cc:205] Tried to move back new op lower bound from 7022889241594101760 to 7022889240366825472. Current Snapshot: MvccSnapshot[applied={T|T < 7022889241594101760}]
INSERT report
rows total: 2000
time total: 122.428 ms
time per row: 0.0612139 ms
I20240501 14:09:28.832315 12444 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:28.837586 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 218f8a4fae324dc7bc535e6a18addb61 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.837908 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218f8a4fae324dc7bc535e6a18addb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.838135 12566 tablet_service.cc:1467] Processing CreateTablet for tablet d178c3291efc44369e3adb921ccc6517 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.838363 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d178c3291efc44369e3adb921ccc6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.839898 14494 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:28.840070 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 218f8a4fae324dc7bc535e6a18addb61 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.840248 12960 tablet_service.cc:1467] Processing CreateTablet for tablet d178c3291efc44369e3adb921ccc6517 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.840314 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218f8a4fae324dc7bc535e6a18addb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.840502 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d178c3291efc44369e3adb921ccc6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.840703 14494 tablet_bootstrap.cc:654] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.841782 14494 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:28.841873 14494 ts_tablet_manager.cc:1367] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.842164 14496 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:28.842326 14495 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:28.842303 14494 raft_consensus.cc:348] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.842398 14494 raft_consensus.cc:374] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.842429 14494 raft_consensus.cc:729] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.842572 14494 consensus_queue.cc:260] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.842916 14494 ts_tablet_manager.cc:1398] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.843021 14496 tablet_bootstrap.cc:654] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.843075 14494 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:28.843119 12829 tablet_service.cc:1467] Processing CreateTablet for tablet d178c3291efc44369e3adb921ccc6517 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.843379 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d178c3291efc44369e3adb921ccc6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.843780 14495 tablet_bootstrap.cc:654] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.843724 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 218f8a4fae324dc7bc535e6a18addb61 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.843880 14494 tablet_bootstrap.cc:654] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.843971 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218f8a4fae324dc7bc535e6a18addb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.844652 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 56197e787ff14e85a07060f23e612498 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.844859 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56197e787ff14e85a07060f23e612498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.845553 14440 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:28.845728 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 56197e787ff14e85a07060f23e612498 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.845942 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56197e787ff14e85a07060f23e612498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.846314 14440 tablet_bootstrap.cc:654] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.847118 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 56197e787ff14e85a07060f23e612498 (DEFAULT_TABLE table=rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:28.847290 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56197e787ff14e85a07060f23e612498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:28.849004 14440 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:28.849093 14440 ts_tablet_manager.cc:1367] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:28.849532 14440 raft_consensus.cc:348] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.849622 14440 raft_consensus.cc:374] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.849655 14440 raft_consensus.cc:729] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.849772 14440 consensus_queue.cc:260] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.850071 14440 ts_tablet_manager.cc:1398] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.850287 14440 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:28.850894 14496 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:28.850981 14496 ts_tablet_manager.cc:1367] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:28.851044 14440 tablet_bootstrap.cc:654] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.851415 14496 raft_consensus.cc:348] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.851521 14496 raft_consensus.cc:374] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.851553 14496 raft_consensus.cc:729] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.851675 14496 consensus_queue.cc:260] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.851930 14496 ts_tablet_manager.cc:1398] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:28.852108 14496 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:28.852217 14494 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:28.852321 14494 ts_tablet_manager.cc:1367] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:28.852866 14494 raft_consensus.cc:348] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.852975 14494 raft_consensus.cc:374] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.853014 14494 raft_consensus.cc:729] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.853150 14494 consensus_queue.cc:260] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.853475 14494 ts_tablet_manager.cc:1398] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.853878 14496 tablet_bootstrap.cc:654] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.854202 14495 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:28.854265 14495 ts_tablet_manager.cc:1367] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:28.854702 14495 raft_consensus.cc:348] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.854835 14495 raft_consensus.cc:374] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.854869 14495 raft_consensus.cc:729] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.854883 14496 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:28.854955 14496 ts_tablet_manager.cc:1367] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:28.854984 14495 consensus_queue.cc:260] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.855280 14495 ts_tablet_manager.cc:1398] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.855367 14440 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:28.855459 14440 ts_tablet_manager.cc:1367] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:28.855844 14440 raft_consensus.cc:348] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.855930 14440 raft_consensus.cc:374] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.855901 14496 raft_consensus.cc:348] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.855970 14440 raft_consensus.cc:729] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.855996 14496 raft_consensus.cc:374] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.856029 14496 raft_consensus.cc:729] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.856072 14440 consensus_queue.cc:260] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.856130 14496 consensus_queue.cc:260] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.856321 14440 ts_tablet_manager.cc:1398] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.856456 14440 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:28.856453 14496 ts_tablet_manager.cc:1398] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.857314 14440 tablet_bootstrap.cc:654] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.858309 14440 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:28.858395 14440 ts_tablet_manager.cc:1367] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.858961 14440 raft_consensus.cc:348] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.859071 14440 raft_consensus.cc:374] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.859097 14440 raft_consensus.cc:729] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.859200 14440 consensus_queue.cc:260] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.859431 14440 ts_tablet_manager.cc:1398] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:28.863408 14495 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:28.864337 14495 tablet_bootstrap.cc:654] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:28.865332 14495 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:28.865429 14495 ts_tablet_manager.cc:1367] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:28.865847 14495 raft_consensus.cc:348] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.865932 14495 raft_consensus.cc:374] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:28.865957 14495 raft_consensus.cc:729] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:28.866055 14495 consensus_queue.cc:260] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.866274 14495 ts_tablet_manager.cc:1398] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:28.876288 14210 raft_consensus.cc:482] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.876363 14210 raft_consensus.cc:504] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.876674 14210 leader_election.cc:288] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:28.876961 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56197e787ff14e85a07060f23e612498" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:28.876992 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56197e787ff14e85a07060f23e612498" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:28.877079 12848 raft_consensus.cc:2454] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:28.877101 12716 raft_consensus.cc:2454] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:28.877363 12913 leader_election.cc:302] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.877532 14210 raft_consensus.cc:2781] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.877588 14210 raft_consensus.cc:482] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.877621 14210 raft_consensus.cc:3037] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.878463 14210 raft_consensus.cc:504] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.878808 14210 leader_election.cc:288] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:28.879055 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56197e787ff14e85a07060f23e612498" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:28.879050 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56197e787ff14e85a07060f23e612498" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:28.879143 12848 raft_consensus.cc:3037] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.879148 12716 raft_consensus.cc:3037] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.879964 12716 raft_consensus.cc:2454] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:28.879992 12848 raft_consensus.cc:2454] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:28.880230 12913 leader_election.cc:302] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.880376 14210 raft_consensus.cc:2781] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.880483 14210 raft_consensus.cc:686] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:28.880594 14210 consensus_queue.cc:237] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:28.882414 12437 catalog_manager.cc:5484] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:28.891062 14548 raft_consensus.cc:482] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:28.891151 14548 raft_consensus.cc:504] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.891472 14548 leader_election.cc:288] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:28.891753 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218f8a4fae324dc7bc535e6a18addb61" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:28.891753 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218f8a4fae324dc7bc535e6a18addb61" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:28.891884 12716 raft_consensus.cc:2454] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:28.891884 12585 raft_consensus.cc:2454] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:28.892134 12782 leader_election.cc:302] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.892292 14548 raft_consensus.cc:2781] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:28.892349 14548 raft_consensus.cc:482] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:28.892376 14548 raft_consensus.cc:3037] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.893179 14548 raft_consensus.cc:504] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.893446 14548 leader_election.cc:288] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:28.893694 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218f8a4fae324dc7bc535e6a18addb61" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:28.893698 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "218f8a4fae324dc7bc535e6a18addb61" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:28.893781 12716 raft_consensus.cc:3037] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.893783 12585 raft_consensus.cc:3037] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:28.894456 12716 raft_consensus.cc:2454] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:28.894775 12782 leader_election.cc:302] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:28.894912 14548 raft_consensus.cc:2781] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:28.895002 14548 raft_consensus.cc:686] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:28.895035 12585 raft_consensus.cc:2454] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:28.895148 14548 consensus_queue.cc:237] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:28.896881 12437 catalog_manager.cc:5484] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:29.020185 14168 raft_consensus.cc:482] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.020318 14168 raft_consensus.cc:504] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.020684 14168 leader_election.cc:288] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:29.021019 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d178c3291efc44369e3adb921ccc6517" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:29.021041 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d178c3291efc44369e3adb921ccc6517" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:29.021152 12978 raft_consensus.cc:2454] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:29.021168 12848 raft_consensus.cc:2454] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:29.021421 12522 leader_election.cc:302] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:29.021600 14168 raft_consensus.cc:2781] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.021659 14168 raft_consensus.cc:482] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.021696 14168 raft_consensus.cc:3037] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.022467 14168 raft_consensus.cc:504] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.022831 14168 leader_election.cc:288] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:29.023101 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d178c3291efc44369e3adb921ccc6517" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:29.023149 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d178c3291efc44369e3adb921ccc6517" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:29.023205 12848 raft_consensus.cc:3037] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.023262 12978 raft_consensus.cc:3037] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.023947 12848 raft_consensus.cc:2454] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:29.023952 12978 raft_consensus.cc:2454] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:29.024201 12523 leader_election.cc:302] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:29.024354 14168 raft_consensus.cc:2781] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.024436 14168 raft_consensus.cc:686] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:29.024575 14168 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.026329 12437 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:29.139892 14563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.140115 14563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.142946 14563 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240501 14:09:29.220351 12716 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.221275 12848 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.221490 12978 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.221786 14459 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.222118 14459 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.222409 14489 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.223891 12848 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.224329 14168 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.266170 12718 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.266517 12584 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.267052 14360 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.267347 14360 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.533 ms
time per row: 0.0592666 ms
I20240501 14:09:29.304006 12437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:29.309381 12566 tablet_service.cc:1467] Processing CreateTablet for tablet e135742313954e889da54e332f689dd7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.309700 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e135742313954e889da54e332f689dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.309767 12827 tablet_service.cc:1467] Processing CreateTablet for tablet e135742313954e889da54e332f689dd7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.309888 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.310029 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e135742313954e889da54e332f689dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.310173 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80afbd22fdad4bcdad3768c4a1953f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.311295 12696 tablet_service.cc:1467] Processing CreateTablet for tablet e135742313954e889da54e332f689dd7 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.311535 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e135742313954e889da54e332f689dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.311987 14494 tablet_bootstrap.cc:492] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:29.312098 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.312141 14440 tablet_bootstrap.cc:492] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:29.312283 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80afbd22fdad4bcdad3768c4a1953f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.312871 14494 tablet_bootstrap.cc:654] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.313038 14440 tablet_bootstrap.cc:654] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.313663 14496 tablet_bootstrap.cc:492] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:29.313663 14495 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:29.314396 14496 tablet_bootstrap.cc:654] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.314395 14495 tablet_bootstrap.cc:654] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.315227 14440 tablet_bootstrap.cc:492] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:29.315304 14440 ts_tablet_manager.cc:1367] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:29.315321 12827 tablet_service.cc:1467] Processing CreateTablet for tablet ca4e8d319cc54e6290d6f9feb54c9794 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.315546 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4e8d319cc54e6290d6f9feb54c9794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.315696 14440 raft_consensus.cc:348] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.315788 14440 raft_consensus.cc:374] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.315815 14440 raft_consensus.cc:729] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.315925 14440 consensus_queue.cc:260] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.315996 14495 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:29.316064 14495 ts_tablet_manager.cc:1367] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:29.316191 14440 ts_tablet_manager.cc:1398] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.316496 14495 raft_consensus.cc:348] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.316596 14495 raft_consensus.cc:374] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.316633 14495 raft_consensus.cc:729] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.316738 14495 consensus_queue.cc:260] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.316996 14495 ts_tablet_manager.cc:1398] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.317353 14440 tablet_bootstrap.cc:492] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:29.318181 14440 tablet_bootstrap.cc:654] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.319950 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.320079 14496 tablet_bootstrap.cc:492] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:29.320140 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80afbd22fdad4bcdad3768c4a1953f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.320086 12565 tablet_service.cc:1467] Processing CreateTablet for tablet ca4e8d319cc54e6290d6f9feb54c9794 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.320163 14496 ts_tablet_manager.cc:1367] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:29.320322 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4e8d319cc54e6290d6f9feb54c9794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.320602 14496 raft_consensus.cc:348] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.320703 14496 raft_consensus.cc:374] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.320730 14496 raft_consensus.cc:729] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.320832 14496 consensus_queue.cc:260] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.321086 14496 ts_tablet_manager.cc:1398] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.321259 14496 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:29.321919 14494 tablet_bootstrap.cc:492] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:29.322016 14494 ts_tablet_manager.cc:1367] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:29.322137 14496 tablet_bootstrap.cc:654] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.322331 14494 raft_consensus.cc:348] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.322422 14494 raft_consensus.cc:374] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.322467 14494 raft_consensus.cc:729] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.322603 14494 consensus_queue.cc:260] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.322870 14494 ts_tablet_manager.cc:1398] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.323027 14494 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:29.323407 12960 tablet_service.cc:1467] Processing CreateTablet for tablet ca4e8d319cc54e6290d6f9feb54c9794 (DEFAULT_TABLE table=rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.323652 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4e8d319cc54e6290d6f9feb54c9794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.323808 14494 tablet_bootstrap.cc:654] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.324774 14494 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:29.324853 14494 ts_tablet_manager.cc:1367] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:29.325198 14495 tablet_bootstrap.cc:492] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:29.325542 14440 tablet_bootstrap.cc:492] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:29.325502 14494 raft_consensus.cc:348] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.325590 14494 raft_consensus.cc:374] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.325615 14440 ts_tablet_manager.cc:1367] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:29.325623 14494 raft_consensus.cc:729] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.325996 14494 consensus_queue.cc:260] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.326005 14440 raft_consensus.cc:348] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.326089 14440 raft_consensus.cc:374] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.326123 14440 raft_consensus.cc:729] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.326179 14495 tablet_bootstrap.cc:654] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.326221 14440 consensus_queue.cc:260] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.326290 14494 ts_tablet_manager.cc:1398] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.326429 14494 tablet_bootstrap.cc:492] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:29.326571 14440 ts_tablet_manager.cc:1398] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.327980 14494 tablet_bootstrap.cc:654] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.330164 14496 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:29.330250 14496 ts_tablet_manager.cc:1367] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:29.330482 14494 tablet_bootstrap.cc:492] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:29.330485 14495 tablet_bootstrap.cc:492] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:29.330545 14494 ts_tablet_manager.cc:1367] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:29.330556 14495 ts_tablet_manager.cc:1367] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:29.330649 14496 raft_consensus.cc:348] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.330785 14496 raft_consensus.cc:374] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.330833 14496 raft_consensus.cc:729] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.330945 14496 consensus_queue.cc:260] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.331022 14495 raft_consensus.cc:348] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.331090 14494 raft_consensus.cc:348] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.331130 14495 raft_consensus.cc:374] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.331171 14494 raft_consensus.cc:374] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.331167 14495 raft_consensus.cc:729] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.331200 14494 raft_consensus.cc:729] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.331225 14496 ts_tablet_manager.cc:1398] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.331341 14494 consensus_queue.cc:260] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.331684 14494 ts_tablet_manager.cc:1398] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.332120 14495 consensus_queue.cc:260] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.332489 14495 ts_tablet_manager.cc:1398] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:29.395843 14252 raft_consensus.cc:482] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.395980 14252 raft_consensus.cc:504] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.396445 14252 leader_election.cc:288] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:29.396848 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135742313954e889da54e332f689dd7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:29.396888 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135742313954e889da54e332f689dd7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:29.396987 12849 raft_consensus.cc:2454] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:29.397043 12584 raft_consensus.cc:2454] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:29.397321 12653 leader_election.cc:302] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:29.397500 14252 raft_consensus.cc:2781] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.397562 14252 raft_consensus.cc:482] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.397614 14252 raft_consensus.cc:3037] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.398417 14252 raft_consensus.cc:504] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.398800 14252 leader_election.cc:288] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:29.400079 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135742313954e889da54e332f689dd7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:29.400193 12585 raft_consensus.cc:3037] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.400969 12585 raft_consensus.cc:2454] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:29.401254 12653 leader_election.cc:302] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:29.401427 14252 raft_consensus.cc:2781] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.401508 14252 raft_consensus.cc:686] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:29.401554 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e135742313954e889da54e332f689dd7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:29.401657 12849 raft_consensus.cc:3037] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.401651 14252 consensus_queue.cc:237] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.402626 12849 raft_consensus.cc:2454] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:29.403569 12437 catalog_manager.cc:5484] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:29.636596 14459 raft_consensus.cc:482] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.636729 14459 raft_consensus.cc:504] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.637144 14459 leader_election.cc:288] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:29.637471 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e8d319cc54e6290d6f9feb54c9794" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:29.637499 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e8d319cc54e6290d6f9feb54c9794" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:29.637602 12585 raft_consensus.cc:2454] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:29.637636 12849 raft_consensus.cc:2454] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:29.637888 12916 leader_election.cc:302] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:29.638075 14459 raft_consensus.cc:2781] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.638140 14459 raft_consensus.cc:482] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.638183 14459 raft_consensus.cc:3037] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.639062 14459 raft_consensus.cc:504] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.639348 14459 leader_election.cc:288] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:29.639688 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e8d319cc54e6290d6f9feb54c9794" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:29.639690 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4e8d319cc54e6290d6f9feb54c9794" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:29.639812 12849 raft_consensus.cc:3037] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.639832 12585 raft_consensus.cc:3037] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.640561 12849 raft_consensus.cc:2454] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:29.640604 12585 raft_consensus.cc:2454] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:29.640866 12916 leader_election.cc:302] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:29.641043 14459 raft_consensus.cc:2781] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.641129 14459 raft_consensus.cc:686] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:29.641284 14459 consensus_queue.cc:237] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.643222 12437 catalog_manager.cc:5484] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:29.667498 14459 raft_consensus.cc:482] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:29.667606 14459 raft_consensus.cc:504] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.667968 14459 leader_election.cc:288] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:29.668334 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80afbd22fdad4bcdad3768c4a1953f0d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:29.668371 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80afbd22fdad4bcdad3768c4a1953f0d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:29.668485 12585 raft_consensus.cc:2454] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:29.668485 12718 raft_consensus.cc:2454] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:29.668761 12913 leader_election.cc:302] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:29.669027 14459 raft_consensus.cc:2781] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:29.669092 14459 raft_consensus.cc:482] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:29.669144 14459 raft_consensus.cc:3037] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.670161 14459 raft_consensus.cc:504] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.670496 14459 leader_election.cc:288] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:29.670802 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80afbd22fdad4bcdad3768c4a1953f0d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:29.670904 12585 raft_consensus.cc:3037] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.670843 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80afbd22fdad4bcdad3768c4a1953f0d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:29.670984 12718 raft_consensus.cc:3037] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:29.671651 12585 raft_consensus.cc:2454] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:29.671743 12718 raft_consensus.cc:2454] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:29.671959 12915 leader_election.cc:302] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:29.672149 14459 raft_consensus.cc:2781] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:29.672219 14459 raft_consensus.cc:686] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:29.672370 14459 consensus_queue.cc:237] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:29.674392 12437 catalog_manager.cc:5484] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:29.795135 14613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:29.795466 14613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:29.798753 14613 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240501 14:09:29.889201 12584 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.889745 14459 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.890905 12847 raft_consensus.cc:1261] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.891070 12585 raft_consensus.cc:1261] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.891556 14452 consensus_queue.cc:1035] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.891575 14210 consensus_queue.cc:1035] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.892171 12718 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.890923 12849 raft_consensus.cc:1261] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.892973 14452 consensus_queue.cc:1035] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.895251 14459 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.895342 12586 raft_consensus.cc:1261] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:29.895733 14210 consensus_queue.cc:1035] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:29.914147 14648 mvcc.cc:205] Tried to move back new op lower bound from 7022889246262730752 to 7022889245379506176. Current Snapshot: MvccSnapshot[applied={T|T < 7022889246262730752}]
I20240501 14:09:29.948066 14650 mvcc.cc:205] Tried to move back new op lower bound from 7022889246286569472 to 7022889245251649536. Current Snapshot: MvccSnapshot[applied={T|T < 7022889246267973632}]
INSERT report
rows total: 2000
time total: 122.794 ms
time per row: 0.0613968 ms
I20240501 14:09:29.965651 12444 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:29.971438 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1c8ff4149c42aa9517af442c38060c (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.971453 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1c8ff4149c42aa9517af442c38060c (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.971767 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1c8ff4149c42aa9517af442c38060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.971768 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1c8ff4149c42aa9517af442c38060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.972344 12959 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8e476bc46420d992a43bf3b4c73fd (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.972344 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1c8ff4149c42aa9517af442c38060c (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.972695 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e476bc46420d992a43bf3b4c73fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.974062 14678 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:29.974989 12565 tablet_service.cc:1467] Processing CreateTablet for tablet d01550698e8449b081a339f6ba30591c (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.975142 14678 tablet_bootstrap.cc:654] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.975296 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01550698e8449b081a339f6ba30591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.975400 14679 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:29.975715 12828 tablet_service.cc:1467] Processing CreateTablet for tablet d01550698e8449b081a339f6ba30591c (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.975934 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01550698e8449b081a339f6ba30591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.975960 12827 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8e476bc46420d992a43bf3b4c73fd (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.976096 14677 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:29.976133 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e476bc46420d992a43bf3b4c73fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.977011 14677 tablet_bootstrap.cc:654] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.978108 14677 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:29.978192 14677 ts_tablet_manager.cc:1367] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.978682 14679 tablet_bootstrap.cc:654] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.978688 14677 raft_consensus.cc:348] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.979024 14677 raft_consensus.cc:374] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.979074 14677 raft_consensus.cc:729] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.979212 14677 consensus_queue.cc:260] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.979533 14677 ts_tablet_manager.cc:1398] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.979650 14679 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:29.979729 14679 ts_tablet_manager.cc:1367] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:29.979729 12698 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8e476bc46420d992a43bf3b4c73fd (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.979949 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e476bc46420d992a43bf3b4c73fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.980181 14679 raft_consensus.cc:348] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.980278 14679 raft_consensus.cc:374] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.980310 14679 raft_consensus.cc:729] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.980445 14679 consensus_queue.cc:260] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.980738 14679 ts_tablet_manager.cc:1398] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.981851 14677 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:29.982020 12696 tablet_service.cc:1467] Processing CreateTablet for tablet d01550698e8449b081a339f6ba30591c (DEFAULT_TABLE table=rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:29.982224 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01550698e8449b081a339f6ba30591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.982672 14677 tablet_bootstrap.cc:654] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.983891 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1c8ff4149c42aa9517af442c38060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:29.984571 14677 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:29.984647 14677 ts_tablet_manager.cc:1367] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:29.985051 14677 raft_consensus.cc:348] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.985160 14677 raft_consensus.cc:374] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.985194 14677 raft_consensus.cc:729] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.985306 14677 consensus_queue.cc:260] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.985569 14681 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:29.985744 14677 ts_tablet_manager.cc:1398] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.985908 14677 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:29.986049 14679 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:29.986871 14677 tablet_bootstrap.cc:654] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.987035 14681 tablet_bootstrap.cc:654] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.987278 14679 tablet_bootstrap.cc:654] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.988008 14677 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:29.988101 14677 ts_tablet_manager.cc:1367] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.988248 14678 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:29.988376 14678 ts_tablet_manager.cc:1367] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:29.988555 14677 raft_consensus.cc:348] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.988639 14677 raft_consensus.cc:374] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.988684 14677 raft_consensus.cc:729] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.988794 14677 consensus_queue.cc:260] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.988827 14678 raft_consensus.cc:348] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.988945 14678 raft_consensus.cc:374] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.988986 14678 raft_consensus.cc:729] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.989048 14677 ts_tablet_manager.cc:1398] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:29.989120 14678 consensus_queue.cc:260] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.989414 14678 ts_tablet_manager.cc:1398] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.989606 14678 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:29.989856 14681 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:29.990005 14681 ts_tablet_manager.cc:1367] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:29.990535 14678 tablet_bootstrap.cc:654] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.990574 14679 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:29.990496 14681 raft_consensus.cc:348] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.990666 14679 ts_tablet_manager.cc:1367] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:29.990780 14681 raft_consensus.cc:374] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.990880 14681 raft_consensus.cc:729] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.991068 14681 consensus_queue.cc:260] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.991111 14679 raft_consensus.cc:348] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.991230 14679 raft_consensus.cc:374] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.991271 14679 raft_consensus.cc:729] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.991398 14679 consensus_queue.cc:260] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:29.991608 14681 ts_tablet_manager.cc:1398] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:29.991706 14679 ts_tablet_manager.cc:1398] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.992527 14678 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:29.992625 14678 ts_tablet_manager.cc:1367] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:29.993047 14678 raft_consensus.cc:348] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.993165 14678 raft_consensus.cc:374] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.993196 14678 raft_consensus.cc:729] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.993306 14678 consensus_queue.cc:260] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.993602 14678 ts_tablet_manager.cc:1398] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:29.993781 14678 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:29.994511 14678 tablet_bootstrap.cc:654] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:29.995586 14678 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:29.995667 14678 ts_tablet_manager.cc:1367] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:29.996104 14678 raft_consensus.cc:348] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.996219 14678 raft_consensus.cc:374] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:29.996245 14678 raft_consensus.cc:729] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:29.996361 14678 consensus_queue.cc:260] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:29.996623 14678 ts_tablet_manager.cc:1398] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.063108 14548 raft_consensus.cc:482] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.063270 14548 raft_consensus.cc:504] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.063732 14548 leader_election.cc:288] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:30.064147 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01550698e8449b081a339f6ba30591c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:30.064170 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01550698e8449b081a339f6ba30591c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:30.064299 12718 raft_consensus.cc:2454] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:30.064322 12585 raft_consensus.cc:2454] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:30.064579 12782 leader_election.cc:302] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:30.064888 14690 raft_consensus.cc:2781] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.064949 14690 raft_consensus.cc:482] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.065006 14690 raft_consensus.cc:3037] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.066095 14690 raft_consensus.cc:504] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.066409 14690 leader_election.cc:288] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:30.066814 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01550698e8449b081a339f6ba30591c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:30.066825 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d01550698e8449b081a339f6ba30591c" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:30.066913 12718 raft_consensus.cc:3037] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.066941 12585 raft_consensus.cc:3037] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.067668 12585 raft_consensus.cc:2454] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:30.067678 12718 raft_consensus.cc:2454] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:30.067972 12784 leader_election.cc:302] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:30.068131 14690 raft_consensus.cc:2781] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.068195 14690 raft_consensus.cc:686] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:30.068327 14690 consensus_queue.cc:237] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.070194 12445 catalog_manager.cc:5484] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:30.172493 14252 raft_consensus.cc:482] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.172668 14252 raft_consensus.cc:504] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.173113 14252 leader_election.cc:288] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:30.173533 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:30.173537 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:30.173693 12978 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:30.173703 12847 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:30.174058 12653 leader_election.cc:302] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:30.174221 14252 raft_consensus.cc:2781] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.174288 14252 raft_consensus.cc:482] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.174342 14252 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.175405 14252 raft_consensus.cc:504] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.175760 14252 leader_election.cc:288] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:30.175974 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:30.176072 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:30.176111 12980 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.176193 12847 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.176893 12980 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:30.177011 12847 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:30.177197 12653 leader_election.cc:302] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:30.177395 14252 raft_consensus.cc:2781] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.177488 14252 raft_consensus.cc:686] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:30.177614 14252 consensus_queue.cc:237] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.179669 12445 catalog_manager.cc:5484] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:30.333810 14252 raft_consensus.cc:482] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.333961 14252 raft_consensus.cc:504] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.334311 14252 leader_election.cc:288] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:30.334712 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1c8ff4149c42aa9517af442c38060c" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:30.334720 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1c8ff4149c42aa9517af442c38060c" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:30.334885 12847 raft_consensus.cc:2454] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:30.334892 12980 raft_consensus.cc:2454] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:30.335247 12653 leader_election.cc:302] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:30.335402 14252 raft_consensus.cc:2781] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.335462 14252 raft_consensus.cc:482] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.335490 14252 raft_consensus.cc:3037] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.336505 14252 raft_consensus.cc:504] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.336833 14252 leader_election.cc:288] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:30.337143 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1c8ff4149c42aa9517af442c38060c" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:30.337245 12847 raft_consensus.cc:3037] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.338014 12847 raft_consensus.cc:2454] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:30.338258 12654 leader_election.cc:302] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:30.338264 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1c8ff4149c42aa9517af442c38060c" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:30.338368 12980 raft_consensus.cc:3037] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.338413 14252 raft_consensus.cc:2781] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.338510 14252 raft_consensus.cc:686] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:30.338893 14252 consensus_queue.cc:237] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.339272 12980 raft_consensus.cc:2454] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:30.341032 12445 catalog_manager.cc:5484] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:30.465780 14694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:30.466089 14694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:30.469350 14694 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 14:09:30.535287 14690 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.539628 14548 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.559373 14252 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.565634 14452 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.573633 12980 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.574409 14452 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.577874 12848 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:30.578608 14252 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:30.599467 14744 mvcc.cc:205] Tried to move back new op lower bound from 7022889249059553280 to 7022889248109436928. Current Snapshot: MvccSnapshot[applied={T|T < 7022889249059553280}]
INSERT report
rows total: 2000
time total: 129.377 ms
time per row: 0.0646887 ms
I20240501 14:09:30.642158 12437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:30.647435 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99b39e354144e48eee75f9f49a09fc (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.647435 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99b39e354144e48eee75f9f49a09fc (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.647725 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99b39e354144e48eee75f9f49a09fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.647725 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99b39e354144e48eee75f9f49a09fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.648232 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99b39e354144e48eee75f9f49a09fc (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.648481 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99b39e354144e48eee75f9f49a09fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.650604 14756 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:30.650708 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 717fa0889b89434ca145e573711b0677 (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.650983 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717fa0889b89434ca145e573711b0677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.651131 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 717fa0889b89434ca145e573711b0677 (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.651160 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 717fa0889b89434ca145e573711b0677 (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.651365 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717fa0889b89434ca145e573711b0677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.651461 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717fa0889b89434ca145e573711b0677. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.651918 14756 tablet_bootstrap.cc:654] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.652526 14755 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:30.653080 14757 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:30.653381 14755 tablet_bootstrap.cc:654] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.653968 14757 tablet_bootstrap.cc:654] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.654448 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.654673 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9334f0f7444909b8e0ff366c71083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.654631 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.654906 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9334f0f7444909b8e0ff366c71083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.655339 14757 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:30.655472 14757 ts_tablet_manager.cc:1367] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.655989 14757 raft_consensus.cc:348] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.656378 14761 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:30.656361 14757 raft_consensus.cc:374] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.656635 14757 raft_consensus.cc:729] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.656754 14756 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:30.656831 14757 consensus_queue.cc:260] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.656917 14756 ts_tablet_manager.cc:1367] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:30.657214 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 (DEFAULT_TABLE table=rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:30.657269 14757 ts_tablet_manager.cc:1398] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.657285 14756 raft_consensus.cc:348] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.657397 14756 raft_consensus.cc:374] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.657445 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9334f0f7444909b8e0ff366c71083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:30.657490 14761 tablet_bootstrap.cc:654] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.657438 14756 raft_consensus.cc:729] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.657611 14756 consensus_queue.cc:260] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.657910 14756 ts_tablet_manager.cc:1398] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.658565 14761 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:30.658651 14761 ts_tablet_manager.cc:1367] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:30.658799 14756 tablet_bootstrap.cc:492] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:30.659193 14761 raft_consensus.cc:348] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.659303 14761 raft_consensus.cc:374] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.659345 14761 raft_consensus.cc:729] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.659469 14761 consensus_queue.cc:260] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.659791 14761 ts_tablet_manager.cc:1398] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:30.660279 14756 tablet_bootstrap.cc:654] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.661434 14756 tablet_bootstrap.cc:492] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:30.661512 14756 ts_tablet_manager.cc:1367] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:30.661592 14757 tablet_bootstrap.cc:492] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:30.661890 14756 raft_consensus.cc:348] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.661993 14756 raft_consensus.cc:374] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.662019 14756 raft_consensus.cc:729] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.662117 14756 consensus_queue.cc:260] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.662386 14756 ts_tablet_manager.cc:1398] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.662545 14756 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:30.663333 14756 tablet_bootstrap.cc:654] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.664312 14755 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:30.664403 14755 ts_tablet_manager.cc:1367] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:30.664835 14755 raft_consensus.cc:348] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.664947 14755 raft_consensus.cc:374] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.665035 14755 raft_consensus.cc:729] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.665055 14757 tablet_bootstrap.cc:654] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.665161 14755 consensus_queue.cc:260] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.665426 14755 ts_tablet_manager.cc:1398] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.665582 14755 tablet_bootstrap.cc:492] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:30.666465 14755 tablet_bootstrap.cc:654] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.667348 14756 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:30.667433 14756 ts_tablet_manager.cc:1367] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:30.667531 14757 tablet_bootstrap.cc:492] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:30.667596 14757 ts_tablet_manager.cc:1367] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:30.667737 14755 tablet_bootstrap.cc:492] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:30.667806 14755 ts_tablet_manager.cc:1367] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.667835 14756 raft_consensus.cc:348] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.667907 14757 raft_consensus.cc:348] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.667943 14756 raft_consensus.cc:374] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.667997 14757 raft_consensus.cc:374] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.668012 14756 raft_consensus.cc:729] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.668025 14757 raft_consensus.cc:729] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.668121 14756 consensus_queue.cc:260] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.668139 14757 consensus_queue.cc:260] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.668380 14756 ts_tablet_manager.cc:1398] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.668397 14757 ts_tablet_manager.cc:1398] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:30.668665 14757 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:30.668231 14755 raft_consensus.cc:348] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.668953 14755 raft_consensus.cc:374] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.669005 14755 raft_consensus.cc:729] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.669456 14757 tablet_bootstrap.cc:654] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:30.669147 14755 consensus_queue.cc:260] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.670455 14755 ts_tablet_manager.cc:1398] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:30.670650 14624 raft_consensus.cc:482] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.670725 14624 raft_consensus.cc:504] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.671033 14757 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:30.671106 14757 ts_tablet_manager.cc:1367] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:30.671113 14624 leader_election.cc:288] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:30.671378 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717fa0889b89434ca145e573711b0677" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:30.671485 12978 raft_consensus.cc:2454] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:30.671500 14757 raft_consensus.cc:348] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.671591 14757 raft_consensus.cc:374] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:30.671744 14757 raft_consensus.cc:729] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:30.671746 12522 leader_election.cc:302] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:30.671898 14624 raft_consensus.cc:2781] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.671964 14624 raft_consensus.cc:482] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.672012 14624 raft_consensus.cc:3037] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.671985 14757 consensus_queue.cc:260] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.672034 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717fa0889b89434ca145e573711b0677" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:30.672147 12718 raft_consensus.cc:2454] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:30.672295 14757 ts_tablet_manager.cc:1398] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:30.672943 14624 raft_consensus.cc:504] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.673290 14624 leader_election.cc:288] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:30.673770 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717fa0889b89434ca145e573711b0677" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:30.673877 12978 raft_consensus.cc:3037] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.675100 12978 raft_consensus.cc:2454] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:30.675387 12522 leader_election.cc:302] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:30.675709 14722 raft_consensus.cc:2781] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.675803 14722 raft_consensus.cc:686] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:30.675920 14722 consensus_queue.cc:237] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:30.676059 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "717fa0889b89434ca145e573711b0677" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:30.676153 12718 raft_consensus.cc:3037] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.677111 12718 raft_consensus.cc:2454] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:30.678155 12442 catalog_manager.cc:5484] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:30.690064 14624 raft_consensus.cc:482] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.690153 14624 raft_consensus.cc:504] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.690479 14624 leader_election.cc:288] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:30.690778 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99b39e354144e48eee75f9f49a09fc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:30.690796 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99b39e354144e48eee75f9f49a09fc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:30.690902 12978 raft_consensus.cc:2454] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:30.690904 12718 raft_consensus.cc:2454] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:30.691136 12522 leader_election.cc:302] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:30.691308 14624 raft_consensus.cc:2781] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.691345 14624 raft_consensus.cc:482] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.691377 14624 raft_consensus.cc:3037] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.692170 14624 raft_consensus.cc:504] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.692438 14624 leader_election.cc:288] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:30.692716 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99b39e354144e48eee75f9f49a09fc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:30.692806 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99b39e354144e48eee75f9f49a09fc" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:30.692833 12718 raft_consensus.cc:3037] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.692909 12978 raft_consensus.cc:3037] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.693763 12978 raft_consensus.cc:2454] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:30.693758 12718 raft_consensus.cc:2454] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:30.694043 12522 leader_election.cc:302] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:30.694219 14624 raft_consensus.cc:2781] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.694296 14624 raft_consensus.cc:686] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:30.694437 14624 consensus_queue.cc:237] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:30.696388 12442 catalog_manager.cc:5484] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:30.860309 14674 raft_consensus.cc:482] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:30.860503 14674 raft_consensus.cc:504] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.860986 14674 leader_election.cc:288] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:30.861303 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9334f0f7444909b8e0ff366c71083" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:30.861444 12585 raft_consensus.cc:2454] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:30.861423 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9334f0f7444909b8e0ff366c71083" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:30.861550 12847 raft_consensus.cc:2454] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:30.861743 12915 leader_election.cc:302] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:30.861953 14674 raft_consensus.cc:2781] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:30.862007 14674 raft_consensus.cc:482] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:30.862047 14674 raft_consensus.cc:3037] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.862937 14674 raft_consensus.cc:504] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.863251 14674 leader_election.cc:288] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:30.863569 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9334f0f7444909b8e0ff366c71083" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:30.863569 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf9334f0f7444909b8e0ff366c71083" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:30.863685 12847 raft_consensus.cc:3037] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.863685 12585 raft_consensus.cc:3037] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:30.864604 12847 raft_consensus.cc:2454] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:30.864605 12585 raft_consensus.cc:2454] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:30.864888 12916 leader_election.cc:302] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:30.865043 14674 raft_consensus.cc:2781] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:30.865264 14674 raft_consensus.cc:686] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:30.865413 14674 consensus_queue.cc:237] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:30.867430 12442 catalog_manager.cc:5484] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:31.002544 14772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.002840 14772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.006237 14772 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 14:09:31.102133 12980 raft_consensus.cc:1261] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.102133 12718 raft_consensus.cc:1261] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.102546 14624 consensus_queue.cc:1035] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.102722 14722 consensus_queue.cc:1035] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.104200 12978 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.104475 12716 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.105147 14624 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.105419 14624 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.109030 12847 raft_consensus.cc:1261] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.109493 14770 consensus_queue.cc:1035] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.110167 12586 raft_consensus.cc:1261] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.111344 14674 consensus_queue.cc:1035] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 138.341 ms
time per row: 0.0691707 ms
I20240501 14:09:31.189663 12445 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:31.194988 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1a9d0e6943dc880161ae320673b6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.195358 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1a9d0e6943dc880161ae320673b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.195467 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3018b4356d462a9f2dd5511392661c (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.195750 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3018b4356d462a9f2dd5511392661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.195722 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3018b4356d462a9f2dd5511392661c (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.195924 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3018b4356d462a9f2dd5511392661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.195922 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 25b5c40793b348bbb0e5251c8d553e30 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.196130 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5c40793b348bbb0e5251c8d553e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.198064 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1a9d0e6943dc880161ae320673b6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.198356 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1a9d0e6943dc880161ae320673b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.198603 14821 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:31.198521 12828 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3018b4356d462a9f2dd5511392661c (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.198521 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1a9d0e6943dc880161ae320673b6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.198922 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3018b4356d462a9f2dd5511392661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.199162 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1a9d0e6943dc880161ae320673b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.199745 14821 tablet_bootstrap.cc:654] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.200740 14823 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:31.200927 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 25b5c40793b348bbb0e5251c8d553e30 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.201426 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5c40793b348bbb0e5251c8d553e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.201650 14823 tablet_bootstrap.cc:654] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.202240 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 25b5c40793b348bbb0e5251c8d553e30 (DEFAULT_TABLE table=rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:31.202435 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5c40793b348bbb0e5251c8d553e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:31.202947 14820 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:31.204370 14820 tablet_bootstrap.cc:654] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.205195 14824 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:31.205288 14823 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:31.205386 14823 ts_tablet_manager.cc:1367] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:31.205430 14821 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:31.205502 14821 ts_tablet_manager.cc:1367] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:31.205804 14823 raft_consensus.cc:348] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.205909 14823 raft_consensus.cc:374] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.205940 14823 raft_consensus.cc:729] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.205880 14821 raft_consensus.cc:348] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.205976 14821 raft_consensus.cc:374] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.206002 14820 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:31.206005 14821 raft_consensus.cc:729] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.206082 14820 ts_tablet_manager.cc:1367] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:31.206056 14823 consensus_queue.cc:260] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.206115 14821 consensus_queue.cc:260] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.206180 14824 tablet_bootstrap.cc:654] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.206326 14823 ts_tablet_manager.cc:1398] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.206354 14821 ts_tablet_manager.cc:1398] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.206491 14823 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:31.206491 14821 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:31.206521 14820 raft_consensus.cc:348] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.206627 14820 raft_consensus.cc:374] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.206661 14820 raft_consensus.cc:729] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.206787 14820 consensus_queue.cc:260] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.207235 14820 ts_tablet_manager.cc:1398] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.207391 14821 tablet_bootstrap.cc:654] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.207513 14823 tablet_bootstrap.cc:654] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.207669 14820 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:31.208617 14820 tablet_bootstrap.cc:654] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.209594 14821 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:31.209772 14820 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:31.209776 14821 ts_tablet_manager.cc:1367] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:31.209939 14820 ts_tablet_manager.cc:1367] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.210358 14821 raft_consensus.cc:348] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.210533 14821 raft_consensus.cc:374] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.210628 14821 raft_consensus.cc:729] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.210608 14820 raft_consensus.cc:348] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.210810 14820 raft_consensus.cc:374] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.210932 14820 raft_consensus.cc:729] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.210923 14821 consensus_queue.cc:260] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.211115 14820 consensus_queue.cc:260] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.211216 14824 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:31.211297 14824 ts_tablet_manager.cc:1367] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:31.211292 14821 ts_tablet_manager.cc:1398] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.211604 14820 ts_tablet_manager.cc:1398] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:31.211720 14824 raft_consensus.cc:348] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.211812 14824 raft_consensus.cc:374] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.211841 14824 raft_consensus.cc:729] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.211946 14824 consensus_queue.cc:260] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.212194 14824 ts_tablet_manager.cc:1398] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.212352 14824 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:31.213152 14824 tablet_bootstrap.cc:654] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.214148 14823 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:31.214207 14823 ts_tablet_manager.cc:1367] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:31.214536 14823 raft_consensus.cc:348] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.214622 14823 raft_consensus.cc:374] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.214651 14823 raft_consensus.cc:729] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.214790 14823 consensus_queue.cc:260] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.215050 14823 ts_tablet_manager.cc:1398] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.216096 14824 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:31.216164 14824 ts_tablet_manager.cc:1367] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:31.216499 14824 raft_consensus.cc:348] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.216580 14824 raft_consensus.cc:374] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.216603 14824 raft_consensus.cc:729] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.216717 14824 consensus_queue.cc:260] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.216943 14824 ts_tablet_manager.cc:1398] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:31.217092 14824 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:31.217844 14824 tablet_bootstrap.cc:654] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:31.218946 14824 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:31.219018 14824 ts_tablet_manager.cc:1367] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:31.219408 14824 raft_consensus.cc:348] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.219496 14824 raft_consensus.cc:374] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:31.219523 14824 raft_consensus.cc:729] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:31.219636 14824 consensus_queue.cc:260] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.219908 14824 ts_tablet_manager.cc:1398] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:31.236049 14674 raft_consensus.cc:482] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:31.236152 14674 raft_consensus.cc:504] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.236626 14674 leader_election.cc:288] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:31.236982 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5c40793b348bbb0e5251c8d553e30" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:31.237005 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5c40793b348bbb0e5251c8d553e30" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:31.237116 12847 raft_consensus.cc:2454] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:31.237164 12717 raft_consensus.cc:2454] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:31.237470 12916 leader_election.cc:302] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:31.237638 14674 raft_consensus.cc:2781] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:31.237691 14674 raft_consensus.cc:482] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:31.237744 14674 raft_consensus.cc:3037] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.238714 14674 raft_consensus.cc:504] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.239094 14674 leader_election.cc:288] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:31.239403 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5c40793b348bbb0e5251c8d553e30" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:31.239516 12847 raft_consensus.cc:3037] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.239504 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b5c40793b348bbb0e5251c8d553e30" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:31.239630 12717 raft_consensus.cc:3037] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.240397 12847 raft_consensus.cc:2454] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:31.240608 12717 raft_consensus.cc:2454] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:31.240907 12916 leader_election.cc:302] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:31.241080 14674 raft_consensus.cc:2781] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:31.241163 14674 raft_consensus.cc:686] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:31.241295 14674 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:31.243314 12445 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:31.308844 14814 raft_consensus.cc:482] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:31.308972 14814 raft_consensus.cc:504] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.309413 14814 leader_election.cc:288] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:31.309671 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3018b4356d462a9f2dd5511392661c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:31.309705 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3018b4356d462a9f2dd5511392661c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:31.309804 12980 raft_consensus.cc:2454] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:31.309819 12847 raft_consensus.cc:2454] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:31.310119 12522 leader_election.cc:302] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:31.310286 14814 raft_consensus.cc:2781] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:31.310334 14624 raft_consensus.cc:482] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:31.310480 14814 raft_consensus.cc:482] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:31.310501 14624 raft_consensus.cc:504] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.310585 14814 raft_consensus.cc:3037] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.310853 14624 leader_election.cc:288] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:31.311201 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:31.311152 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:31.311312 12847 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:31.311316 12717 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:31.311655 12520 leader_election.cc:302] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:31.311829 14814 raft_consensus.cc:504] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.311899 14624 raft_consensus.cc:2781] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:31.311944 14624 raft_consensus.cc:482] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:31.311985 14624 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.312381 14814 leader_election.cc:288] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:31.312763 12847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3018b4356d462a9f2dd5511392661c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:31.312891 12847 raft_consensus.cc:3037] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.312929 14624 raft_consensus.cc:504] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.313254 14624 leader_election.cc:288] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:31.313587 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:31.313591 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3018b4356d462a9f2dd5511392661c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:31.313707 12980 raft_consensus.cc:3037] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.313704 12849 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.313786 12847 raft_consensus.cc:2454] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:31.314455 12849 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:31.314716 12980 raft_consensus.cc:2454] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:31.314944 12523 leader_election.cc:302] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:31.315105 14624 raft_consensus.cc:2781] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:31.315133 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:31.315160 12523 leader_election.cc:302] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:31.315194 14624 raft_consensus.cc:686] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:31.315281 14814 raft_consensus.cc:2781] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:31.315299 12717 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:31.315377 14814 raft_consensus.cc:686] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:31.315331 14624 consensus_queue.cc:237] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:31.315523 14814 consensus_queue.cc:237] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:31.316391 12717 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:31.317626 12445 catalog_manager.cc:5484] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:31.317870 12445 catalog_manager.cc:5484] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:31.441942 14837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.442209 14837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.445621 14837 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 14:09:31.535434 12978 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.535694 12849 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.536304 14722 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.536442 14814 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.547987 12718 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.547987 12717 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.548404 12848 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.548877 14722 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.551281 12849 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:31.551529 14814 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.553401 14674 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:31.554175 14674 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.278 ms
time per row: 0.0626392 ms
I20240501 14:09:31.614799 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.5:33177
--local_ip_for_outbound_sockets=127.6.82.5
--tserver_master_addrs=127.6.82.62:35905
--webserver_port=38729
--webserver_interface=127.6.82.5
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:09:31.735646 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.735883 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:31.735953 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.739248 14876 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:31.739331 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:31.739430 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.5
I20240501 14:09:31.742970 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.5:33177
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.82.5
--webserver_port=38729
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.14876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.5
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:31.743842 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:31.744935 14876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:31.752951 14882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.753100 14876 server_base.cc:1034] running on GCE node
W20240501 14:09:31.752960 14884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.753268 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.753675 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:31.754170 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:31.755332 14876 hybrid_clock.cc:648] HybridClock initialized: now 1714572571755313 us; error 28 us; skew 500 ppm
I20240501 14:09:31.763825 14876 webserver.cc:460] Webserver started at http://127.6.82.5:38729/ using document root <none> and password file <none>
I20240501 14:09:31.764394 14876 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:31.764462 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:31.768043 14876 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:31.769964 14890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.770946 14876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.771039 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8a13c028ba8a4a659b74af6c265df95e"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:31.771425 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:31.784706 14876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:31.785212 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:31.785341 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:31.785812 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:31.786679 14876 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:31.786751 14876 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.786808 14876 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:31.786844 14876 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.795230 14876 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.5:33177
I20240501 14:09:31.795251 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.5:33177 every 8 connection(s)
I20240501 14:09:31.796269 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:09:31.800680 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 14876
I20240501 14:09:31.801076 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.6:37673
--local_ip_for_outbound_sockets=127.6.82.6
--tserver_master_addrs=127.6.82.62:35905
--webserver_port=33199
--webserver_interface=127.6.82.6
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:31.805426 15004 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:31.805687 15004 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:31.806391 15004 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:31.807669 12442 ts_manager.cc:194] Re-registered known tserver with Master: 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:31.808336 12442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.5:54315
W20240501 14:09:31.921272 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:31.921483 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:31.921541 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:31.924718 15007 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:31.924824 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:31.924927 15007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.6
I20240501 14:09:31.928893 15007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.6:37673
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.82.6
--webserver_port=33199
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.15007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.6
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:31.929735 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:31.930935 15007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:31.939221 15016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:31.939221 15014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.939383 15007 server_base.cc:1034] running on GCE node
W20240501 14:09:31.939488 15013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:31.939879 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:31.940367 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:31.941511 15007 hybrid_clock.cc:648] HybridClock initialized: now 1714572571941495 us; error 30 us; skew 500 ppm
I20240501 14:09:31.943356 15007 webserver.cc:460] Webserver started at http://127.6.82.6:33199/ using document root <none> and password file <none>
I20240501 14:09:31.943895 15007 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:31.943972 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:31.947230 15007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:31.949247 15022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.950199 15007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:31.950311 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ab90e77f0027449bb3155980ca95593c"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:31.950814 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:31.981709 15007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:31.982416 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:31.982573 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:31.983161 15007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:31.984011 15007 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:31.984068 15007 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.984110 15007 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:31.984140 15007 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:31.992964 15007 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.6:37673
I20240501 14:09:31.993005 15135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.6:37673 every 8 connection(s)
I20240501 14:09:31.994006 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:09:31.997721 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 15007
I20240501 14:09:31.998142 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.7:34847
--local_ip_for_outbound_sockets=127.6.82.7
--tserver_master_addrs=127.6.82.62:35905
--webserver_port=44773
--webserver_interface=127.6.82.7
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:32.003667 15136 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:32.003919 15136 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:32.004683 15136 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:32.005750 12442 ts_manager.cc:194] Re-registered known tserver with Master: ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:32.006297 12442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.6:55505
W20240501 14:09:32.116156 15139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:32.116449 15139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:32.116571 15139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:32.119742 15139 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:32.119805 15139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:32.119910 15139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.7
I20240501 14:09:32.123773 15139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.7:34847
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.82.7
--webserver_port=44773
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.15139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.7
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:32.124965 15139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:32.126196 15139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:32.133685 15145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:32.133679 15147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:32.133914 15139 server_base.cc:1034] running on GCE node
W20240501 14:09:32.133980 15144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:32.134506 15139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:32.135110 15139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:32.136338 15139 hybrid_clock.cc:648] HybridClock initialized: now 1714572572136313 us; error 38 us; skew 500 ppm
I20240501 14:09:32.138434 15139 webserver.cc:460] Webserver started at http://127.6.82.7:44773/ using document root <none> and password file <none>
I20240501 14:09:32.139006 15139 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:32.139075 15139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:32.142184 15139 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:32.143975 15153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.144878 15139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:32.144986 15139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "66e2dc34330848c3b8c2dd15898d34c8"
format_stamp: "Formatted at 2024-05-01 14:09:16 on dist-test-slave-crgf"
I20240501 14:09:32.145361 15139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:32.163573 15139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:32.164184 15139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:32.164342 15139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:32.164907 15139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:32.165823 15139 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:32.165871 15139 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.165918 15139 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:32.165949 15139 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.174136 15139 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.7:34847
I20240501 14:09:32.174181 15267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.7:34847 every 8 connection(s)
I20240501 14:09:32.175181 15139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:09:32.184021 15268 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:32.184190 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 15139
I20240501 14:09:32.184230 15268 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:32.184561 6472 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
/tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.82.8:46801
--local_ip_for_outbound_sockets=127.6.82.8
--tserver_master_addrs=127.6.82.62:35905
--webserver_port=41901
--webserver_interface=127.6.82.8
--builtin_ntp_servers=127.6.82.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:09:32.184756 15268 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:32.187403 12442 ts_manager.cc:194] Re-registered known tserver with Master: 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:32.187969 12442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.7:37117
W20240501 14:09:32.302007 15271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:09:32.302229 15271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:09:32.302300 15271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:09:32.305645 15271 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:09:32.305722 15271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:09:32.305828 15271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.82.8
I20240501 14:09:32.309629 15271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.82.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.82.8:46801
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.82.8
--webserver_port=41901
--tserver_master_addrs=127.6.82.62:35905
--never_fsync=true
--heap_profile_path=/tmp/kudu.15271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.82.8
--log_dir=/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 13:43:12 UTC on 1a15035a6830
build id 73
I20240501 14:09:32.310387 15271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:09:32.311446 15271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:09:32.318991 15279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:32.319173 15271 server_base.cc:1034] running on GCE node
W20240501 14:09:32.319237 15277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:09:32.318989 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:09:32.319651 15271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:09:32.320183 15271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:09:32.321401 15271 hybrid_clock.cc:648] HybridClock initialized: now 1714572572321384 us; error 26 us; skew 500 ppm
I20240501 14:09:32.323529 15271 webserver.cc:460] Webserver started at http://127.6.82.8:41901/ using document root <none> and password file <none>
I20240501 14:09:32.324203 15271 fs_manager.cc:359] Metadata directory not provided
I20240501 14:09:32.324297 15271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:09:32.327600 15271 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:09:32.329514 15285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.330441 15271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:32.330538 15271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f23f426391724d6688aafffe7ee017d8"
format_stamp: "Formatted at 2024-05-01 14:09:17 on dist-test-slave-crgf"
I20240501 14:09:32.330972 15271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:09:32.344596 15271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:09:32.345085 15271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:09:32.345228 15271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:09:32.345818 15271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:09:32.346704 15271 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:09:32.346798 15271 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.346869 15271 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:09:32.346899 15271 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:09:32.355381 15271 rpc_server.cc:307] RPC server started. Bound to: 127.6.82.8:46801
I20240501 14:09:32.355398 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.82.8:46801 every 8 connection(s)
I20240501 14:09:32.356410 15271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:09:32.361261 6472 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrG4Tzf/build/debug/bin/kudu as pid 15271
I20240501 14:09:32.364604 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:32.366786 15399 heartbeater.cc:347] Connected to a master server at 127.6.82.62:35905
I20240501 14:09:32.367066 15399 heartbeater.cc:464] Registering TS with master...
I20240501 14:09:32.367794 15399 heartbeater.cc:511] Master 127.6.82.62:35905 requested a full tablet report, sending...
I20240501 14:09:32.379016 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 537b2cc4b6b34826af9eb6450111553b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3]), partition=
I20240501 14:09:32.379307 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b2cc4b6b34826af9eb6450111553b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.386251 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 537b2cc4b6b34826af9eb6450111553b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3]), partition=
I20240501 14:09:32.387044 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b2cc4b6b34826af9eb6450111553b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.392388 15415 tablet_bootstrap.cc:492] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:32.392617 12442 ts_manager.cc:194] Re-registered known tserver with Master: f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:32.393323 12442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.82.8:41429
I20240501 14:09:32.393627 15416 tablet_bootstrap.cc:492] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:32.394765 15415 tablet_bootstrap.cc:654] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.395834 15415 tablet_bootstrap.cc:492] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:32.395974 15415 ts_tablet_manager.cc:1367] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:32.396467 15415 raft_consensus.cc:348] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.396700 15415 raft_consensus.cc:374] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.396806 15415 raft_consensus.cc:729] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.397017 15415 consensus_queue.cc:260] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.397472 15415 ts_tablet_manager.cc:1398] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.398108 15416 tablet_bootstrap.cc:654] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.399019 15416 log.cc:824] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.407500 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 537b2cc4b6b34826af9eb6450111553b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3]), partition=
I20240501 14:09:32.408198 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b2cc4b6b34826af9eb6450111553b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.413453 15419 tablet_bootstrap.cc:492] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:32.416149 15416 tablet_bootstrap.cc:492] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:32.416368 15416 ts_tablet_manager.cc:1367] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240501 14:09:32.419257 15419 tablet_bootstrap.cc:654] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.419586 15416 raft_consensus.cc:348] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.419837 15416 raft_consensus.cc:374] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.419929 15416 raft_consensus.cc:729] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.420256 15419 log.cc:824] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.420396 15416 consensus_queue.cc:260] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.421232 15416 ts_tablet_manager.cc:1398] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:32.422363 15419 tablet_bootstrap.cc:492] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:32.422425 15004 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:32.422582 15419 ts_tablet_manager.cc:1367] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 14:09:32.426266 15419 raft_consensus.cc:348] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.426483 15419 raft_consensus.cc:374] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.426537 15419 raft_consensus.cc:729] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.427203 15419 consensus_queue.cc:260] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.427860 15136 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:32.428090 15419 ts_tablet_manager.cc:1398] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240501 14:09:32.431249 15422 raft_consensus.cc:482] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.431407 15422 raft_consensus.cc:504] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.433214 15422 leader_election.cc:288] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:32.436805 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b2cc4b6b34826af9eb6450111553b" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:32.436970 12586 raft_consensus.cc:2454] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:32.437403 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b2cc4b6b34826af9eb6450111553b" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:32.437651 14958 raft_consensus.cc:2454] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:32.438190 15024 leader_election.cc:302] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:32.438426 15422 raft_consensus.cc:2781] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:32.438484 15422 raft_consensus.cc:482] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:32.438540 15422 raft_consensus.cc:3037] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.439435 15422 raft_consensus.cc:504] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.439798 15422 leader_election.cc:288] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:32.440109 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b2cc4b6b34826af9eb6450111553b" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:32.440155 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b2cc4b6b34826af9eb6450111553b" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:32.440222 12586 raft_consensus.cc:3037] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.440271 14958 raft_consensus.cc:3037] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.440917 12586 raft_consensus.cc:2454] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:32.441221 15025 leader_election.cc:302] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:32.441306 14958 raft_consensus.cc:2454] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:32.441383 15422 raft_consensus.cc:2781] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:32.441613 15422 raft_consensus.cc:686] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:32.441972 15422 consensus_queue.cc:237] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:32.445282 12443 catalog_manager.cc:5484] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:32.494993 15137 tablet.cc:2369] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:32.547483 15005 tablet.cc:2369] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:32.560091 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:32.566133 12586 raft_consensus.cc:1261] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:32.566656 15425 consensus_queue.cc:1035] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:32.566949 14958 raft_consensus.cc:1261] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:32.567541 15425 consensus_queue.cc:1035] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:32.569991 15448 mvcc.cc:205] Tried to move back new op lower bound from 7022889257225392128 to 7022889256728207360. Current Snapshot: MvccSnapshot[applied={T|T < 7022889257225392128}]
I20240501 14:09:32.569999 15470 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:32.571215 15472 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:32.571844 15470 tablet.cc:1620] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:32.571902 15472 tablet.cc:1620] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:32.572441 12444 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 537b2cc4b6b34826af9eb6450111553b (table rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3]). Expected version 1 got 0
I20240501 14:09:32.574357 12441 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 537b2cc4b6b34826af9eb6450111553b (table rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3]). Expected version 1 got 0
I20240501 14:09:32.577605 15471 mvcc.cc:205] Tried to move back new op lower bound from 7022889257225392128 to 7022889256728207360. Current Snapshot: MvccSnapshot[applied={T|T < 7022889257225392128}]
I20240501 14:09:32.578121 15474 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:32.580812 12444 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 537b2cc4b6b34826af9eb6450111553b (table rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3]). Expected version 1 got 0
I20240501 14:09:32.581334 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3] alter complete (version 1)
I20240501 14:09:32.581182 15474 tablet.cc:1620] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:32.652439 14938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.652951 12829 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.655835 12565 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:32.666158 12696 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:32.669631 15070 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:32.670750 15333 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.686018 15202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.687467 12445 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:32.694851 15470 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:32.695186 15470 tablet.cc:1620] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:32.695233 15474 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:32.695487 15474 tablet.cc:1620] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:32.699277 12959 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:32.700594 15472 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:32.700892 15472 tablet.cc:1620] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:32.705060 12445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=bfe9b781eea44916bd81f71ebce0dfa3] alter complete (version 2)
I20240501 14:09:32.807144 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:32.817435 15470 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:32.819913 15474 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:32.822387 15472 tablet.cc:1722] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:32.828703 12445 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=bfe9b781eea44916bd81f71ebce0dfa3] alter complete (version 3)
I20240501 14:09:32.852012 12829 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:32.858148 15333 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.859687 12959 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:32.860800 12696 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:32.862988 12436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:32.866896 15202 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.871979 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 75e7c78545bf45be968aa742ac30603e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.871999 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd29fa5dc2b4408878ceb3c12a57098 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.872268 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e7c78545bf45be968aa742ac30603e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.872268 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd29fa5dc2b4408878ceb3c12a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.872351 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 75e7c78545bf45be968aa742ac30603e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.872452 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd29fa5dc2b4408878ceb3c12a57098 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.872707 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e7c78545bf45be968aa742ac30603e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.872725 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd29fa5dc2b4408878ceb3c12a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.874977 15416 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:32.875022 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 41a1a3342cae4af480f1e65164493219 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.875149 15419 tablet_bootstrap.cc:492] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:32.875228 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a1a3342cae4af480f1e65164493219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.875926 15416 tablet_bootstrap.cc:654] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.876331 15419 tablet_bootstrap.cc:654] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.879029 12436 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:09:32.879173 12436 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 14:09:32.881953 14938 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:32.885811 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 497040dcb4334bcd95fe5931f2b953f5
I20240501 14:09:32.886196 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 547dce5435754bea84a0c68aa415e228
I20240501 14:09:32.886417 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet bebdcf50ed0046a29f4fbfac71087214
I20240501 14:09:32.887132 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet 497040dcb4334bcd95fe5931f2b953f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:32.887213 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 547dce5435754bea84a0c68aa415e228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:32.888110 15553 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:32.885792 15552 tablet_bootstrap.cc:492] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:32.888137 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 5dd29fa5dc2b4408878ceb3c12a57098 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.889142 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd29fa5dc2b4408878ceb3c12a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.889724 15416 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:32.889811 15416 ts_tablet_manager.cc:1367] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:32.890337 15416 raft_consensus.cc:348] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.890453 15416 raft_consensus.cc:374] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.890482 15416 raft_consensus.cc:729] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.890592 15416 consensus_queue.cc:260] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.890841 15416 ts_tablet_manager.cc:1398] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.890851 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet bebdcf50ed0046a29f4fbfac71087214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:32.891232 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 41a1a3342cae4af480f1e65164493219 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.891459 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a1a3342cae4af480f1e65164493219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.892446 15552 tablet_bootstrap.cc:654] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.892608 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 547dce5435754bea84a0c68aa415e228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:32.892908 15553 tablet_bootstrap.cc:654] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.893972 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet bebdcf50ed0046a29f4fbfac71087214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:32.893972 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 497040dcb4334bcd95fe5931f2b953f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:32.894368 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet bebdcf50ed0046a29f4fbfac71087214 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:32.894949 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 41a1a3342cae4af480f1e65164493219 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.895864 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a1a3342cae4af480f1e65164493219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.898053 15333 tablet_service.cc:1467] Processing CreateTablet for tablet 75e7c78545bf45be968aa742ac30603e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:32.898309 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75e7c78545bf45be968aa742ac30603e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:32.894786 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 497040dcb4334bcd95fe5931f2b953f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:32.900084 15553 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:32.900264 15553 ts_tablet_manager.cc:1367] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:32.900681 15553 raft_consensus.cc:348] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.900875 15553 raft_consensus.cc:374] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.900954 15553 raft_consensus.cc:729] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.901145 15553 consensus_queue.cc:260] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.901717 15553 ts_tablet_manager.cc:1398] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.904177 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 547dce5435754bea84a0c68aa415e228 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:32.904311 15556 tablet_replica.cc:330] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:32.907720 12567 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:32.905683 15552 tablet_bootstrap.cc:492] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:32.908234 15552 ts_tablet_manager.cc:1367] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:32.908692 15552 raft_consensus.cc:348] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.908875 15552 raft_consensus.cc:374] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.908946 15552 raft_consensus.cc:729] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.909111 15552 consensus_queue.cc:260] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.909173 15556 raft_consensus.cc:2229] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.909474 15556 raft_consensus.cc:2258] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.910210 15552 ts_tablet_manager.cc:1398] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:32.910466 15556 ts_tablet_manager.cc:1875] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.912786 15556 ts_tablet_manager.cc:1888] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.914954 15556 log.cc:1196] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/547dce5435754bea84a0c68aa415e228
I20240501 14:09:32.915378 15556 ts_tablet_manager.cc:1909] T 547dce5435754bea84a0c68aa415e228 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:32.916293 15556 tablet_replica.cc:330] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:32.916383 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 547dce5435754bea84a0c68aa415e228 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.916529 15556 raft_consensus.cc:2229] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.916697 15556 raft_consensus.cc:2258] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.917192 15556 ts_tablet_manager.cc:1875] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.918871 15556 ts_tablet_manager.cc:1888] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.918994 15556 log.cc:1196] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/497040dcb4334bcd95fe5931f2b953f5
I20240501 14:09:32.919093 15559 tablet_replica.cc:330] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:32.919396 15556 ts_tablet_manager.cc:1909] T 497040dcb4334bcd95fe5931f2b953f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:32.919966 15556 tablet_replica.cc:330] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:32.920034 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 497040dcb4334bcd95fe5931f2b953f5 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.920260 15556 raft_consensus.cc:2229] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.920569 15556 raft_consensus.cc:2258] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.920950 15557 tablet_replica.cc:330] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:32.921253 15557 raft_consensus.cc:2229] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.921552 15557 raft_consensus.cc:2258] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.922444 15563 tablet_bootstrap.cc:492] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:32.922526 15557 ts_tablet_manager.cc:1875] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.924577 15557 ts_tablet_manager.cc:1888] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.924643 15557 log.cc:1196] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/547dce5435754bea84a0c68aa415e228
I20240501 14:09:32.924891 15557 ts_tablet_manager.cc:1909] T 547dce5435754bea84a0c68aa415e228 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:32.919440 15559 raft_consensus.cc:2229] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.925258 15421 raft_consensus.cc:482] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:32.925384 15421 raft_consensus.cc:504] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.926452 15421 leader_election.cc:288] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:32.927604 15559 raft_consensus.cc:2258] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.928859 15562 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:32.929700 15559 ts_tablet_manager.cc:1875] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.931699 12436 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:32.931818 12436 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:32.933719 15559 ts_tablet_manager.cc:1888] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.934056 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 537b2cc4b6b34826af9eb6450111553b
I20240501 14:09:32.934677 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 537b2cc4b6b34826af9eb6450111553b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:32.934782 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 537b2cc4b6b34826af9eb6450111553b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:32.935142 15070 tablet_service.cc:1514] Processing DeleteTablet for tablet 537b2cc4b6b34826af9eb6450111553b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:32 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:32.935722 15563 tablet_bootstrap.cc:654] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.937527 15557 tablet_replica.cc:330] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:32.937747 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 547dce5435754bea84a0c68aa415e228 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.938007 15557 raft_consensus.cc:2229] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.938023 15562 tablet_bootstrap.cc:654] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.938550 15563 log.cc:824] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.939419 15557 raft_consensus.cc:2258] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.940004 15557 ts_tablet_manager.cc:1875] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.940593 15562 log.cc:824] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:09:32.940770 15070 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:32.942273 15556 ts_tablet_manager.cc:1875] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.944236 15559 log.cc:1196] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/bebdcf50ed0046a29f4fbfac71087214
I20240501 14:09:32.944272 15557 ts_tablet_manager.cc:1888] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.944412 15557 log.cc:1196] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/497040dcb4334bcd95fe5931f2b953f5
I20240501 14:09:32.944721 15557 ts_tablet_manager.cc:1909] T 497040dcb4334bcd95fe5931f2b953f5 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:32.944965 15559 ts_tablet_manager.cc:1909] T bebdcf50ed0046a29f4fbfac71087214 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:32.946062 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet bebdcf50ed0046a29f4fbfac71087214 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.946278 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 497040dcb4334bcd95fe5931f2b953f5 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.946552 15556 ts_tablet_manager.cc:1888] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.946573 15567 tablet_replica.cc:330] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:32.946630 15556 log.cc:1196] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/bebdcf50ed0046a29f4fbfac71087214
I20240501 14:09:32.946904 15556 ts_tablet_manager.cc:1909] T bebdcf50ed0046a29f4fbfac71087214 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:32.947046 15567 raft_consensus.cc:2229] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:32.947444 15556 tablet_replica.cc:330] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:32.947527 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet bebdcf50ed0046a29f4fbfac71087214 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.947589 15556 raft_consensus.cc:2229] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.947685 15556 raft_consensus.cc:2258] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.947989 15559 tablet_replica.cc:330] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:32.948130 15567 raft_consensus.cc:2258] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.948199 15566 tablet_replica.cc:330] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:32.948333 15559 raft_consensus.cc:2229] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.948470 15559 raft_consensus.cc:2258] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.948762 15566 raft_consensus.cc:2229] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.948987 15559 ts_tablet_manager.cc:1875] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.948339 15419 tablet_bootstrap.cc:492] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:32.949898 15419 ts_tablet_manager.cc:1367] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.001s
I20240501 14:09:32.950425 15419 raft_consensus.cc:348] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.950554 15419 raft_consensus.cc:374] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.950542 15566 raft_consensus.cc:2258] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.950582 15419 raft_consensus.cc:729] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.950686 15419 consensus_queue.cc:260] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.950937 15419 ts_tablet_manager.cc:1398] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.951092 15419 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:32.951416 15556 ts_tablet_manager.cc:1875] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.952179 15419 tablet_bootstrap.cc:654] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.951093 15567 ts_tablet_manager.cc:1875] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.954512 15558 tablet_replica.cc:330] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:32.954829 15558 raft_consensus.cc:2229] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.955205 15563 tablet_bootstrap.cc:492] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:32.955390 15563 ts_tablet_manager.cc:1367] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240501 14:09:32.955504 15558 raft_consensus.cc:2258] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.955878 15566 ts_tablet_manager.cc:1875] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.957875 15558 ts_tablet_manager.cc:1875] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.959272 15556 ts_tablet_manager.cc:1888] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:32.959349 15556 log.cc:1196] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/537b2cc4b6b34826af9eb6450111553b
I20240501 14:09:32.960223 15567 ts_tablet_manager.cc:1888] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:32.961159 15567 log.cc:1196] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/537b2cc4b6b34826af9eb6450111553b
I20240501 14:09:32.961602 15567 ts_tablet_manager.cc:1909] T 537b2cc4b6b34826af9eb6450111553b P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:32.962582 15562 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:32.962917 15562 ts_tablet_manager.cc:1367] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 14:09:32.962864 15419 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:32.963537 15419 ts_tablet_manager.cc:1367] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:32.963918 15419 raft_consensus.cc:348] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.963996 15419 raft_consensus.cc:374] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.964017 15419 raft_consensus.cc:729] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.964151 15419 consensus_queue.cc:260] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.964387 15419 ts_tablet_manager.cc:1398] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:32.966594 15558 ts_tablet_manager.cc:1888] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.966894 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd29fa5dc2b4408878ceb3c12a57098" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:32.967034 12848 raft_consensus.cc:2454] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:32.967417 14894 leader_election.cc:302] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:32.967501 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 537b2cc4b6b34826af9eb6450111553b (table rebalancer_dynamic_table_new_name [id=bfe9b781eea44916bd81f71ebce0dfa3]) successfully deleted
I20240501 14:09:32.967665 15421 raft_consensus.cc:2781] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:32.967737 15421 raft_consensus.cc:482] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:32.967788 15421 raft_consensus.cc:3037] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.968501 15563 raft_consensus.cc:348] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.968739 15563 raft_consensus.cc:374] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.968763 15421 raft_consensus.cc:504] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.968816 15563 raft_consensus.cc:729] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.969151 15421 leader_election.cc:288] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:32.969265 15563 consensus_queue.cc:260] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.969951 15399 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:32.970136 15563 ts_tablet_manager.cc:1398] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:09:32.970502 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd29fa5dc2b4408878ceb3c12a57098" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:32.970609 12848 raft_consensus.cc:3037] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:32.970667 15559 ts_tablet_manager.cc:1888] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.970775 15559 log.cc:1196] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/547dce5435754bea84a0c68aa415e228
I20240501 14:09:32.971084 15559 ts_tablet_manager.cc:1909] T 547dce5435754bea84a0c68aa415e228 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:32.971522 12848 raft_consensus.cc:2454] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:32.971868 15556 ts_tablet_manager.cc:1909] T 537b2cc4b6b34826af9eb6450111553b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:32.972067 15558 log.cc:1196] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/bebdcf50ed0046a29f4fbfac71087214
I20240501 14:09:32.972512 15558 ts_tablet_manager.cc:1909] T bebdcf50ed0046a29f4fbfac71087214 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:32.972787 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd29fa5dc2b4408878ceb3c12a57098" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:32.972698 15562 raft_consensus.cc:348] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.972878 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 537b2cc4b6b34826af9eb6450111553b (table rebalancer_dynamic_table_new_name [id=bfe9b781eea44916bd81f71ebce0dfa3]) successfully deleted
I20240501 14:09:32.972918 15562 raft_consensus.cc:374] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.972961 15562 raft_consensus.cc:729] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.973335 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd29fa5dc2b4408878ceb3c12a57098" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:32.973510 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 547dce5435754bea84a0c68aa415e228 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.973979 15562 consensus_queue.cc:260] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
W20240501 14:09:32.974059 14891 leader_election.cc:341] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:09:32.974380 14891 leader_election.cc:341] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:32.974869 15558 tablet_replica.cc:330] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:32.974937 15562 ts_tablet_manager.cc:1398] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240501 14:09:32.975271 15268 heartbeater.cc:503] Master 127.6.82.62:35905 was elected leader, sending a full tablet report...
I20240501 14:09:32.970399 15563 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:32.975106 15558 raft_consensus.cc:2229] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:32.976390 15558 raft_consensus.cc:2258] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:32.977236 15562 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:32.977339 15566 ts_tablet_manager.cc:1888] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:32.977456 15566 log.cc:1196] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/537b2cc4b6b34826af9eb6450111553b
I20240501 14:09:32.977838 15566 ts_tablet_manager.cc:1909] T 537b2cc4b6b34826af9eb6450111553b P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:32.978056 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet bebdcf50ed0046a29f4fbfac71087214 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.978127 15558 ts_tablet_manager.cc:1875] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:32.978401 15562 tablet_bootstrap.cc:654] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.978590 15563 tablet_bootstrap.cc:654] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:32.979197 14894 leader_election.cc:302] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters: 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:32.979472 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 537b2cc4b6b34826af9eb6450111553b (table rebalancer_dynamic_table_new_name [id=bfe9b781eea44916bd81f71ebce0dfa3]) successfully deleted
I20240501 14:09:32.979651 15421 raft_consensus.cc:2781] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:32.979821 15421 raft_consensus.cc:686] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:32.980041 15421 consensus_queue.cc:237] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.981043 15563 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:32.981122 15563 ts_tablet_manager.cc:1367] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:32.981621 15563 raft_consensus.cc:348] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.981715 15563 raft_consensus.cc:374] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.981746 15563 raft_consensus.cc:729] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.981861 15563 consensus_queue.cc:260] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:32.982131 15563 ts_tablet_manager.cc:1398] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:32.982609 15558 ts_tablet_manager.cc:1888] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:32.982744 15558 log.cc:1196] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/497040dcb4334bcd95fe5931f2b953f5
I20240501 14:09:32.982919 12438 catalog_manager.cc:5484] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:32.985612 15562 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:32.985740 15562 ts_tablet_manager.cc:1367] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:32.986397 15558 ts_tablet_manager.cc:1909] T 497040dcb4334bcd95fe5931f2b953f5 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:32.986598 15562 raft_consensus.cc:348] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.986704 15562 raft_consensus.cc:374] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:32.986754 15562 raft_consensus.cc:729] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:32.986932 15562 consensus_queue.cc:260] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:32.987056 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 497040dcb4334bcd95fe5931f2b953f5 (table rebalance_test_table_0 [id=23835187044345ba80f04127bb68977c]) successfully deleted
I20240501 14:09:32.987289 15562 ts_tablet_manager.cc:1398] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.004794 15571 raft_consensus.cc:482] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.005165 15571 raft_consensus.cc:504] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.007481 15571 leader_election.cc:288] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:33.011149 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7c78545bf45be968aa742ac30603e" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:33.011286 12717 raft_consensus.cc:2454] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:33.011613 15286 leader_election.cc:302] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:33.011837 15571 raft_consensus.cc:2781] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.011916 15571 raft_consensus.cc:482] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.011862 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7c78545bf45be968aa742ac30603e" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:33.011952 15571 raft_consensus.cc:3037] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.012050 15090 raft_consensus.cc:2454] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:33.012948 15571 raft_consensus.cc:504] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.013247 15571 leader_election.cc:288] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:33.013571 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7c78545bf45be968aa742ac30603e" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:33.013664 12717 raft_consensus.cc:3037] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.014560 12717 raft_consensus.cc:2454] T 75e7c78545bf45be968aa742ac30603e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:33.014853 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e7c78545bf45be968aa742ac30603e" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:33.014883 15286 leader_election.cc:302] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:33.014961 15090 raft_consensus.cc:3037] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.015128 15571 raft_consensus.cc:2781] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.015319 15571 raft_consensus.cc:686] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:33.015749 15090 raft_consensus.cc:2454] T 75e7c78545bf45be968aa742ac30603e P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:33.015797 15571 consensus_queue.cc:237] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.020393 12436 catalog_manager.cc:5484] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:33.036418 12436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:33.041957 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3cfeb3b1654d4e97335eb835d4219a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]), partition=
I20240501 14:09:33.042132 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3cfeb3b1654d4e97335eb835d4219a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]), partition=
I20240501 14:09:33.042239 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3cfeb3b1654d4e97335eb835d4219a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.042351 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3cfeb3b1654d4e97335eb835d4219a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.043139 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3cfeb3b1654d4e97335eb835d4219a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]), partition=
I20240501 14:09:33.043326 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3cfeb3b1654d4e97335eb835d4219a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.044454 15563 tablet_bootstrap.cc:492] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:33.044641 15562 tablet_bootstrap.cc:492] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.045080 15419 tablet_bootstrap.cc:492] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:33.045382 15563 tablet_bootstrap.cc:654] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.045425 15562 tablet_bootstrap.cc:654] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.045892 15419 tablet_bootstrap.cc:654] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.047057 15419 tablet_bootstrap.cc:492] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:33.047158 15419 ts_tablet_manager.cc:1367] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:33.047719 15562 tablet_bootstrap.cc:492] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:33.047719 15563 tablet_bootstrap.cc:492] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:33.047791 15562 ts_tablet_manager.cc:1367] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:33.047791 15563 ts_tablet_manager.cc:1367] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:33.047751 15419 raft_consensus.cc:348] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.047842 15419 raft_consensus.cc:374] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.047873 15419 raft_consensus.cc:729] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.047989 15419 consensus_queue.cc:260] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.048225 15419 ts_tablet_manager.cc:1398] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.048207 15562 raft_consensus.cc:348] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.048207 15563 raft_consensus.cc:348] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.048301 15562 raft_consensus.cc:374] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.048316 15563 raft_consensus.cc:374] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.048338 15562 raft_consensus.cc:729] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.048358 15563 raft_consensus.cc:729] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.048466 15562 consensus_queue.cc:260] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.048466 15563 consensus_queue.cc:260] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.048694 15562 ts_tablet_manager.cc:1398] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.048694 15563 ts_tablet_manager.cc:1398] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.060298 15425 raft_consensus.cc:482] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.060420 15425 raft_consensus.cc:504] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.060978 15425 leader_election.cc:288] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:33.065469 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3cfeb3b1654d4e97335eb835d4219a" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:33.065744 15221 raft_consensus.cc:2454] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:33.065855 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3cfeb3b1654d4e97335eb835d4219a" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:33.066049 15353 raft_consensus.cc:2454] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:33.066311 15023 leader_election.cc:302] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:33.066671 15425 raft_consensus.cc:2781] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.066766 15425 raft_consensus.cc:482] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.066823 15425 raft_consensus.cc:3037] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.067566 15425 raft_consensus.cc:504] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.067853 15425 leader_election.cc:288] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:33.068192 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3cfeb3b1654d4e97335eb835d4219a" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:33.068315 15221 raft_consensus.cc:3037] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.068194 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3cfeb3b1654d4e97335eb835d4219a" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:33.068713 15353 raft_consensus.cc:3037] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.069746 15353 raft_consensus.cc:2454] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:33.069801 15221 raft_consensus.cc:2454] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:33.070116 15026 leader_election.cc:302] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:33.070267 15425 raft_consensus.cc:2781] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.070338 15425 raft_consensus.cc:686] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:33.070482 15425 consensus_queue.cc:237] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:33.072192 12438 catalog_manager.cc:5484] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:33.107604 15400 tablet.cc:2369] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:33.129200 15572 raft_consensus.cc:482] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.129319 15572 raft_consensus.cc:504] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.130290 15572 leader_election.cc:288] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:33.134517 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a1a3342cae4af480f1e65164493219" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:33.134698 15353 raft_consensus.cc:2454] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:33.134876 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a1a3342cae4af480f1e65164493219" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:33.134994 15090 raft_consensus.cc:2454] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:33.135357 15156 leader_election.cc:302] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:33.135846 15572 raft_consensus.cc:2781] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.135915 15572 raft_consensus.cc:482] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.135959 15572 raft_consensus.cc:3037] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.136915 15572 raft_consensus.cc:504] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.137210 15572 leader_election.cc:288] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:33.137538 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a1a3342cae4af480f1e65164493219" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:33.137635 15090 raft_consensus.cc:3037] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.138342 15090 raft_consensus.cc:2454] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:33.138631 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a1a3342cae4af480f1e65164493219" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:33.138687 15156 leader_election.cc:302] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:33.138772 15353 raft_consensus.cc:3037] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.138892 15572 raft_consensus.cc:2781] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.139091 15572 raft_consensus.cc:686] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:33.139601 15353 raft_consensus.cc:2454] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:33.139636 15572 consensus_queue.cc:237] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.142743 12437 catalog_manager.cc:5484] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.167403 15201 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.168068 14938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.168766 12829 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:33.173316 12567 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240501 14:09:33.178992 15269 tablet.cc:2369] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:33.179615 12437 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:33.184037 15221 raft_consensus.cc:1261] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.184490 15422 consensus_queue.cc:1035] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.185696 15353 raft_consensus.cc:1261] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.187800 15422 consensus_queue.cc:1035] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.189707 12440 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]). Expected version 1 got 0
I20240501 14:09:33.190855 12440 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]). Expected version 1 got 0
I20240501 14:09:33.194598 15332 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.195932 15070 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.196653 12696 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 14:09:33.197929 12959 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:33.208057 15470 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.208384 15470 tablet.cc:1620] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:33.211999 12437 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]). Expected version 1 got 0
I20240501 14:09:33.221846 12437 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018] alter complete (version 1)
I20240501 14:09:33.222093 15643 mvcc.cc:205] Tried to move back new op lower bound from 7022889259757625344 to 7022889259298148352. Current Snapshot: MvccSnapshot[applied={T|T < 7022889259757625344}]
I20240501 14:09:33.233382 15647 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.234503 15647 tablet.cc:1620] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:33.241586 15648 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.242118 12437 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018]). Expected version 1 got 0
I20240501 14:09:33.242522 15648 tablet.cc:1620] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:33.284045 15572 raft_consensus.cc:482] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.284191 15572 raft_consensus.cc:504] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.284713 15572 leader_election.cc:288] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:33.287997 12586 consensus_queue.cc:237] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.288760 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd29fa5dc2b4408878ceb3c12a57098" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:33.289125 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dd29fa5dc2b4408878ceb3c12a57098" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:33.289304 12849 raft_consensus.cc:2379] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66e2dc34330848c3b8c2dd15898d34c8 in current term 1: Already voted for candidate 8a13c028ba8a4a659b74af6c265df95e in this term.
I20240501 14:09:33.289619 15157 leader_election.cc:302] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8; no voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:33.289783 15572 raft_consensus.cc:3037] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.290032 12717 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.290392 12849 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.290474 15655 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.290848 15572 raft_consensus.cc:2726] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:33.290827 15476 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.292826 14722 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.293211 12849 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.294198 12717 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.296126 12437 catalog_manager.cc:5484] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:33.297348 12521 consensus_peers.cc:485] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66436017fd044f0b919629933ea56e9c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.306699 12717 consensus_queue.cc:237] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.308490 12978 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.308629 12586 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.308923 14452 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.309101 15665 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.311091 14452 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.311714 12586 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.311667 12978 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
W20240501 14:09:33.313194 12653 consensus_peers.cc:485] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6305f3f7cb8846bf843d2d753fd1ae07. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.315062 12440 catalog_manager.cc:5484] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.323299 12586 consensus_queue.cc:237] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.324703 12978 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.325168 15476 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.325536 12444 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:33.326057 12849 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.326925 15476 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.328047 15476 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.328840 12978 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.329514 12849 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.333015 12438 catalog_manager.cc:5484] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.333655 15470 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
W20240501 14:09:33.333807 12523 consensus_peers.cc:485] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b547fcb34f12402abccdb23678c95e26. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.333966 15470 tablet.cc:1620] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:33.334797 15647 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:33.335101 15647 tablet.cc:1620] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:33.337044 15648 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:33.337363 15648 tablet.cc:1620] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:33.341271 12444 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=541d0cb27f8747ce9b0205744823f018] alter complete (version 2)
I20240501 14:09:33.347405 12978 consensus_queue.cc:237] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:33.349184 12717 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.349624 15683 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.350117 12849 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.350581 15683 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.351560 15684 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.351995 12717 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.353055 12849 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
W20240501 14:09:33.354879 12913 consensus_peers.cc:485] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8147fae0de98429ab6554c7e7f5bf8ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.356030 12444 catalog_manager.cc:5484] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.364280 12586 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
W20240501 14:09:33.365799 12521 consensus_peers.cc:485] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d178c3291efc44369e3adb921ccc6517. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.365832 12978 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.365940 12849 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.366226 15476 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.366427 14722 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.368319 14722 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.368744 12978 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.370493 12849 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.373260 12444 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.381335 12586 consensus_queue.cc:237] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.382829 12717 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.383384 12978 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.383884 14722 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.384081 15476 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.385475 15573 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.386791 14722 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.387388 12717 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
W20240501 14:09:33.388499 12522 consensus_peers.cc:485] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afa5ebe09c99435fb69ceb527ab4734a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.388912 12978 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.390520 12440 catalog_manager.cc:5484] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.393395 12437 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:09:33.393527 12437 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 14:09:33.396085 12437 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 5b30c755056741dda7ab43f07aaf2609
I20240501 14:09:33.396390 12437 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet cbeba1e7c473403da87b5a0cd3c70d7e
I20240501 14:09:33.396644 12437 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet eca34c8d867f4f2293ad5aac5fb8aadb
I20240501 14:09:33.396661 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b30c755056741dda7ab43f07aaf2609 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:33.396767 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b30c755056741dda7ab43f07aaf2609 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:33.396883 15556 tablet_replica.cc:330] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:33.396929 15558 tablet_replica.cc:330] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:33.397030 15556 raft_consensus.cc:2229] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.397050 15558 raft_consensus.cc:2229] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.397178 15558 raft_consensus.cc:2258] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.397186 15556 raft_consensus.cc:2258] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.397717 15558 ts_tablet_manager.cc:1875] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.397727 15556 ts_tablet_manager.cc:1875] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.398032 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b30c755056741dda7ab43f07aaf2609 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:33.398190 15559 tablet_replica.cc:330] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:33.398298 15559 raft_consensus.cc:2229] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.398471 15559 raft_consensus.cc:2258] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.399048 15559 ts_tablet_manager.cc:1875] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.399971 15556 ts_tablet_manager.cc:1888] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.400038 15556 log.cc:1196] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/5b30c755056741dda7ab43f07aaf2609
I20240501 14:09:33.400031 15558 ts_tablet_manager.cc:1888] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.400102 15558 log.cc:1196] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b30c755056741dda7ab43f07aaf2609
I20240501 14:09:33.400297 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet cbeba1e7c473403da87b5a0cd3c70d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:33.400329 15556 ts_tablet_manager.cc:1909] T 5b30c755056741dda7ab43f07aaf2609 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:33.400674 15557 tablet_replica.cc:330] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:33.400830 15557 raft_consensus.cc:2229] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.400964 15557 raft_consensus.cc:2258] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.401113 15559 ts_tablet_manager.cc:1888] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.401161 15559 log.cc:1196] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/5b30c755056741dda7ab43f07aaf2609
I20240501 14:09:33.401367 15559 ts_tablet_manager.cc:1909] T 5b30c755056741dda7ab43f07aaf2609 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:33.401471 15557 ts_tablet_manager.cc:1875] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.398041 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet eca34c8d867f4f2293ad5aac5fb8aadb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:33.402024 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet cbeba1e7c473403da87b5a0cd3c70d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:33.402086 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet eca34c8d867f4f2293ad5aac5fb8aadb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:33.402108 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 5b30c755056741dda7ab43f07aaf2609 (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.402180 15556 tablet_replica.cc:330] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:33.402328 15556 raft_consensus.cc:2229] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.402516 15556 raft_consensus.cc:2258] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.402580 15559 tablet_replica.cc:330] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:33.402673 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 5b30c755056741dda7ab43f07aaf2609 (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.402701 15559 raft_consensus.cc:2229] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.402853 15559 raft_consensus.cc:2258] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.403045 15556 ts_tablet_manager.cc:1875] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.403254 15559 ts_tablet_manager.cc:1875] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.403688 15557 ts_tablet_manager.cc:1888] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.403769 15557 log.cc:1196] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/cbeba1e7c473403da87b5a0cd3c70d7e
I20240501 14:09:33.404069 15557 ts_tablet_manager.cc:1909] T cbeba1e7c473403da87b5a0cd3c70d7e P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:33.404958 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet cbeba1e7c473403da87b5a0cd3c70d7e (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.405027 15556 ts_tablet_manager.cc:1888] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.405099 15556 log.cc:1196] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/cbeba1e7c473403da87b5a0cd3c70d7e
I20240501 14:09:33.405383 15556 ts_tablet_manager.cc:1909] T cbeba1e7c473403da87b5a0cd3c70d7e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:33.405444 15559 ts_tablet_manager.cc:1888] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.405592 15559 log.cc:1196] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/eca34c8d867f4f2293ad5aac5fb8aadb
I20240501 14:09:33.405822 15559 ts_tablet_manager.cc:1909] T eca34c8d867f4f2293ad5aac5fb8aadb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:33.405957 15556 tablet_replica.cc:330] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:33.406057 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet cbeba1e7c473403da87b5a0cd3c70d7e (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.406147 15556 raft_consensus.cc:2229] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.406298 15556 raft_consensus.cc:2258] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.406453 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet eca34c8d867f4f2293ad5aac5fb8aadb (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.406813 15556 ts_tablet_manager.cc:1875] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.407096 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet cbeba1e7c473403da87b5a0cd3c70d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:33.407123 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet eca34c8d867f4f2293ad5aac5fb8aadb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:33.407591 15573 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.408021 15558 ts_tablet_manager.cc:1909] T 5b30c755056741dda7ab43f07aaf2609 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:33.408645 15571 consensus_queue.cc:1035] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.408771 15558 tablet_replica.cc:330] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:33.408885 15556 ts_tablet_manager.cc:1888] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.408898 15558 raft_consensus.cc:2229] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.408962 15556 log.cc:1196] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/eca34c8d867f4f2293ad5aac5fb8aadb
I20240501 14:09:33.409009 15558 raft_consensus.cc:2258] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.409265 15556 ts_tablet_manager.cc:1909] T eca34c8d867f4f2293ad5aac5fb8aadb P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:33.409492 15558 ts_tablet_manager.cc:1875] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.409924 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 5b30c755056741dda7ab43f07aaf2609 (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.411444 15558 ts_tablet_manager.cc:1888] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.411511 15558 log.cc:1196] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/cbeba1e7c473403da87b5a0cd3c70d7e
I20240501 14:09:33.411865 15558 ts_tablet_manager.cc:1909] T cbeba1e7c473403da87b5a0cd3c70d7e P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:33.412192 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet eca34c8d867f4f2293ad5aac5fb8aadb (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.412525 15558 tablet_replica.cc:330] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:33.412623 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet cbeba1e7c473403da87b5a0cd3c70d7e (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.412650 15558 raft_consensus.cc:2229] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.413353 15558 raft_consensus.cc:2258] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.413815 15558 ts_tablet_manager.cc:1875] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.415666 15558 ts_tablet_manager.cc:1888] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.416388 15558 log.cc:1196] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/eca34c8d867f4f2293ad5aac5fb8aadb
I20240501 14:09:33.416879 15558 ts_tablet_manager.cc:1909] T eca34c8d867f4f2293ad5aac5fb8aadb P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:33.418265 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet eca34c8d867f4f2293ad5aac5fb8aadb (table rebalance_test_table_2 [id=b99f7e18b5ae494b9816eecd1902e1ab]) successfully deleted
I20240501 14:09:33.424110 15590 consensus_queue.cc:1035] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.430616 12978 consensus_queue.cc:237] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.432015 12717 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.432588 12586 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.432579 15683 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.433511 15683 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.434332 14674 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
W20240501 14:09:33.435461 12916 consensus_peers.cc:485] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80afbd22fdad4bcdad3768c4a1953f0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.435812 12586 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.436945 12717 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.437266 12437 catalog_manager.cc:5484] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.443893 12437 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:33.446507 12586 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:33.448094 12849 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.448949 15696 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.450354 12717 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.450713 15470 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:33.451205 15647 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:33.452625 15696 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:33.453617 12520 consensus_peers.cc:485] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7693e3e204634676a758726bb6d3d2ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.454594 15696 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.456231 15648 tablet.cc:1722] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:33.457710 12717 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.458283 12849 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.458825 12443 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.459264 12437 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=541d0cb27f8747ce9b0205744823f018] alter complete (version 3)
I20240501 14:09:33.469205 12717 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.470544 12849 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.470959 12978 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.470996 15665 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.471335 14452 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.473542 15665 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.473938 12849 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
W20240501 14:09:33.474526 12652 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abac52c109174021ac11360b0d22ccfa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.476125 12978 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.477382 12437 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.483974 12717 consensus_queue.cc:237] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.485236 12849 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.485297 12978 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:33.485580 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e13e55ae52a949bc893de94082130157. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.485755 15008 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.485958 14452 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.487804 14452 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.488106 12978 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.489252 12849 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.490607 12443 catalog_manager.cc:5484] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.498185 12717 consensus_queue.cc:237] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.499657 12980 raft_consensus.cc:1261] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.499812 12586 raft_consensus.cc:1261] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.500202 15665 consensus_queue.cc:1035] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.500382 14452 consensus_queue.cc:1035] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.502103 15665 raft_consensus.cc:2932] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.502904 12980 raft_consensus.cc:2932] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
W20240501 14:09:33.503145 12654 consensus_peers.cc:485] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b152ca89c5a416f9b1a65217e4a950e. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.503521 12586 raft_consensus.cc:2932] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.505824 12437 catalog_manager.cc:5484] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.511878 12980 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.513307 12849 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.513703 15683 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.513782 12717 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.514171 15683 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.515570 14674 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.516152 12849 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.516472 12717 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
W20240501 14:09:33.517985 12914 consensus_peers.cc:485] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63a76e89d58d48ba9d1b3b98ef6c172d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.519131 12438 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.527477 12980 consensus_queue.cc:237] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:33.528632 12586 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.528699 12849 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:33.528935 12913 consensus_peers.cc:485] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49f2b0e67d3c43d584ac7583b98caeb9. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.529059 15683 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.529239 14674 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.530879 15684 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.531914 12849 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.534638 12443 catalog_manager.cc:5484] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.535864 12586 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:33.565290 12437 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:33.565464 12437 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:33.567909 12437 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 1d3cfeb3b1654d4e97335eb835d4219a
I20240501 14:09:33.568691 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3cfeb3b1654d4e97335eb835d4219a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:33.568804 15070 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3cfeb3b1654d4e97335eb835d4219a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:33.569270 15779 tablet_replica.cc:330] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:33.569242 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3cfeb3b1654d4e97335eb835d4219a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:33.569609 15779 raft_consensus.cc:2229] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.569664 15780 tablet_replica.cc:330] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:33.569798 15779 raft_consensus.cc:2258] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.569902 15780 raft_consensus.cc:2229] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.570034 15780 raft_consensus.cc:2258] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.570904 15779 ts_tablet_manager.cc:1875] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.571139 15780 ts_tablet_manager.cc:1875] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.571864 15778 tablet_replica.cc:330] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:33.572078 15778 raft_consensus.cc:2229] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.572702 15778 raft_consensus.cc:2258] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.573298 15779 ts_tablet_manager.cc:1888] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.573298 15780 ts_tablet_manager.cc:1888] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.573390 15780 log.cc:1196] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/1d3cfeb3b1654d4e97335eb835d4219a
I20240501 14:09:33.573426 15779 log.cc:1196] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/1d3cfeb3b1654d4e97335eb835d4219a
I20240501 14:09:33.573741 15780 ts_tablet_manager.cc:1909] T 1d3cfeb3b1654d4e97335eb835d4219a P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:33.573802 15779 ts_tablet_manager.cc:1909] T 1d3cfeb3b1654d4e97335eb835d4219a P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:33.574136 15778 ts_tablet_manager.cc:1875] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.574985 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table_new_name [id=541d0cb27f8747ce9b0205744823f018]) successfully deleted
I20240501 14:09:33.575196 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table_new_name [id=541d0cb27f8747ce9b0205744823f018]) successfully deleted
I20240501 14:09:33.576886 15778 ts_tablet_manager.cc:1888] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.576963 15778 log.cc:1196] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/1d3cfeb3b1654d4e97335eb835d4219a
I20240501 14:09:33.577277 15778 ts_tablet_manager.cc:1909] T 1d3cfeb3b1654d4e97335eb835d4219a P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:33.577913 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 1d3cfeb3b1654d4e97335eb835d4219a (table rebalancer_dynamic_table_new_name [id=541d0cb27f8747ce9b0205744823f018]) successfully deleted
I20240501 14:09:33.644448 12829 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:33.645052 14938 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.646575 15201 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.651700 12960 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:33.653275 15070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:33.658006 15572 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.664582 12565 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:33.664902 15572 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.665544 15332 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:33.673847 12437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:33.682472 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63]), partition=
I20240501 14:09:33.682983 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63]), partition=
I20240501 14:09:33.683202 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.683205 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63]), partition=
I20240501 14:09:33.683411 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.682722 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.692373 12437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:33.698962 15822 tablet_bootstrap.cc:492] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:33.699843 15822 tablet_bootstrap.cc:654] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.700138 15821 tablet_bootstrap.cc:492] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.700902 12698 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 14:09:33.702034 15070 tablet_service.cc:1467] Processing CreateTablet for tablet df57dee35f924801a23bd9b5c547a55f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.702363 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df57dee35f924801a23bd9b5c547a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.702488 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab31238e6b4ea4b0da2561c79e7efd (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.702816 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab31238e6b4ea4b0da2561c79e7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.702941 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab31238e6b4ea4b0da2561c79e7efd (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.703183 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab31238e6b4ea4b0da2561c79e7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.705227 15333 tablet_service.cc:1467] Processing CreateTablet for tablet dea946f39526401eb9b684f60cd9e8d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.705313 15202 tablet_service.cc:1467] Processing CreateTablet for tablet df57dee35f924801a23bd9b5c547a55f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.705480 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea946f39526401eb9b684f60cd9e8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.705518 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df57dee35f924801a23bd9b5c547a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.707513 15200 tablet_service.cc:1467] Processing CreateTablet for tablet dea946f39526401eb9b684f60cd9e8d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.707644 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab31238e6b4ea4b0da2561c79e7efd (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.707787 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea946f39526401eb9b684f60cd9e8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.707883 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab31238e6b4ea4b0da2561c79e7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.709900 14937 tablet_service.cc:1467] Processing CreateTablet for tablet dea946f39526401eb9b684f60cd9e8d0 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.710179 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea946f39526401eb9b684f60cd9e8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.710474 14938 tablet_service.cc:1467] Processing CreateTablet for tablet df57dee35f924801a23bd9b5c547a55f (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:33.710660 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df57dee35f924801a23bd9b5c547a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.710669 15821 tablet_bootstrap.cc:654] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.711568 15822 tablet_bootstrap.cc:492] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:33.711639 15822 ts_tablet_manager.cc:1367] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:33.712009 15822 raft_consensus.cc:348] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.713627 15821 tablet_bootstrap.cc:492] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:33.713717 15821 ts_tablet_manager.cc:1367] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:33.712251 15822 raft_consensus.cc:374] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.714080 15822 raft_consensus.cc:729] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.714180 15821 raft_consensus.cc:348] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.714288 15821 raft_consensus.cc:374] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.714236 15822 consensus_queue.cc:260] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.714324 15821 raft_consensus.cc:729] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.714479 15821 consensus_queue.cc:260] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.714879 15821 ts_tablet_manager.cc:1398] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.715034 15821 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.716034 15821 tablet_bootstrap.cc:654] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.717015 15821 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:33.717087 15821 ts_tablet_manager.cc:1367] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.717339 15823 tablet_bootstrap.cc:492] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:33.717522 15821 raft_consensus.cc:348] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.717624 15821 raft_consensus.cc:374] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.717662 15821 raft_consensus.cc:729] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.717777 15821 consensus_queue.cc:260] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.717821 15822 ts_tablet_manager.cc:1398] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:33.718021 15821 ts_tablet_manager.cc:1398] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.718185 15821 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.718597 15824 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:33.719055 15821 tablet_bootstrap.cc:654] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.719648 15824 tablet_bootstrap.cc:654] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.722389 15825 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:33.722573 15821 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:33.722648 15821 ts_tablet_manager.cc:1367] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:33.723140 15821 raft_consensus.cc:348] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.723237 15821 raft_consensus.cc:374] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.723274 15821 raft_consensus.cc:729] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.721041 15823 tablet_bootstrap.cc:654] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.723394 15821 consensus_queue.cc:260] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.723659 15821 ts_tablet_manager.cc:1398] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.723834 15821 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.724205 15824 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:33.724248 15825 tablet_bootstrap.cc:654] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.724284 15824 ts_tablet_manager.cc:1367] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:33.724694 15821 tablet_bootstrap.cc:654] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.724730 15824 raft_consensus.cc:348] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.724826 15824 raft_consensus.cc:374] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.724875 15824 raft_consensus.cc:729] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.724997 15824 consensus_queue.cc:260] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.725303 15824 ts_tablet_manager.cc:1398] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.725471 15824 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:33.726372 15824 tablet_bootstrap.cc:654] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.726691 15833 ts_tablet_manager.cc:902] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:33.728576 15833 tablet_copy_client.cc:308] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:33.729122 15821 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:33.729197 15821 ts_tablet_manager.cc:1367] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:33.729523 15821 raft_consensus.cc:348] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.729617 15821 raft_consensus.cc:374] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.729650 15821 raft_consensus.cc:729] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.729770 15821 consensus_queue.cc:260] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.730039 15821 ts_tablet_manager.cc:1398] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.730679 15824 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:33.730778 15824 ts_tablet_manager.cc:1367] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:33.731262 15824 raft_consensus.cc:348] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.731351 15824 raft_consensus.cc:374] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.731386 15824 raft_consensus.cc:729] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.731489 15824 consensus_queue.cc:260] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.731732 15824 ts_tablet_manager.cc:1398] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.731885 15824 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:33.732717 15824 tablet_bootstrap.cc:654] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.734907 15824 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:33.734999 15824 ts_tablet_manager.cc:1367] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:33.735466 15824 raft_consensus.cc:348] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.735582 15824 raft_consensus.cc:374] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.735621 15824 raft_consensus.cc:729] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.735574 15823 tablet_bootstrap.cc:492] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:33.735783 15823 ts_tablet_manager.cc:1367] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:33.735744 15824 consensus_queue.cc:260] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.736006 15824 ts_tablet_manager.cc:1398] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.736536 15823 raft_consensus.cc:348] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.736657 15823 raft_consensus.cc:374] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.736696 15823 raft_consensus.cc:729] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.736907 15823 consensus_queue.cc:260] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.737754 15823 ts_tablet_manager.cc:1398] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.737955 15823 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:33.739091 15823 tablet_bootstrap.cc:654] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.740113 15823 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:33.740196 15823 ts_tablet_manager.cc:1367] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.740706 15823 raft_consensus.cc:348] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.740969 15823 raft_consensus.cc:374] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.741026 15823 raft_consensus.cc:729] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.741150 15823 consensus_queue.cc:260] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.741575 15823 ts_tablet_manager.cc:1398] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.743969 15573 raft_consensus.cc:482] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.744052 15573 raft_consensus.cc:504] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.744486 15573 leader_election.cc:288] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:33.744925 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:33.745064 15221 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:33.735699 15825 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:33.745339 15825 ts_tablet_manager.cc:1367] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:33.745543 14891 leader_election.cc:302] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:33.745708 15573 raft_consensus.cc:2781] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.745764 15573 raft_consensus.cc:482] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.745807 15573 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.745865 15825 raft_consensus.cc:348] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.746402 15422 raft_consensus.cc:482] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.746482 15422 raft_consensus.cc:504] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.746838 15573 raft_consensus.cc:504] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.746968 15422 leader_election.cc:288] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:33.747161 15573 leader_election.cc:288] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:33.747190 15825 raft_consensus.cc:374] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.747248 15825 raft_consensus.cc:729] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.747403 15825 consensus_queue.cc:260] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.747511 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d5f20c67fc48aeacc0c9b0a9b26a55" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:33.747642 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:33.747743 15222 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.748679 15222 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:33.748936 15825 ts_tablet_manager.cc:1398] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:33.749102 15825 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:33.747627 15221 raft_consensus.cc:2454] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:33.749488 15023 leader_election.cc:302] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:33.749522 14891 leader_election.cc:302] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:33.749754 15573 raft_consensus.cc:2781] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.749850 15573 raft_consensus.cc:686] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:33.749941 15422 raft_consensus.cc:2781] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.749989 15422 raft_consensus.cc:482] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.750025 15422 raft_consensus.cc:3037] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.749984 15573 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.750092 15825 tablet_bootstrap.cc:654] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:33.750684 15421 raft_consensus.cc:482] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.751052 15422 raft_consensus.cc:504] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.751371 15422 leader_election.cc:288] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:33.751339 15421 raft_consensus.cc:504] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.751720 15421 leader_election.cc:288] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:33.751785 12438 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:33.753125 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d5f20c67fc48aeacc0c9b0a9b26a55" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:33.753219 15221 raft_consensus.cc:3037] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.754027 15221 raft_consensus.cc:2454] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:33.754323 15023 leader_election.cc:302] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:33.754491 15422 raft_consensus.cc:2781] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.754563 15422 raft_consensus.cc:686] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:33.754716 15422 consensus_queue.cc:237] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:33.754801 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea946f39526401eb9b684f60cd9e8d0" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:33.754889 15221 raft_consensus.cc:2454] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:33.755234 14891 leader_election.cc:302] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:33.755363 15825 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:33.755443 15421 raft_consensus.cc:2781] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.755481 15825 ts_tablet_manager.cc:1367] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:33.755506 15421 raft_consensus.cc:482] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.755549 15421 raft_consensus.cc:3037] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.755937 15825 raft_consensus.cc:348] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.756048 15825 raft_consensus.cc:374] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:33.756099 15825 raft_consensus.cc:729] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:33.756222 15825 consensus_queue.cc:260] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.756402 15421 raft_consensus.cc:504] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.756721 15421 leader_election.cc:288] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:33.756731 12437 catalog_manager.cc:5484] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.757584 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea946f39526401eb9b684f60cd9e8d0" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:33.757669 15221 raft_consensus.cc:3037] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.758183 15825 ts_tablet_manager.cc:1398] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:33.758479 15221 raft_consensus.cc:2454] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:33.758924 14891 leader_election.cc:302] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:33.759465 15421 raft_consensus.cc:2781] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.759548 15421 raft_consensus.cc:686] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:33.759687 15421 consensus_queue.cc:237] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:33.761492 12437 catalog_manager.cc:5484] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:33.762724 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:33.762864 15090 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:33.763093 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:33.763195 15089 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.764160 15089 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:33.765709 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 66436017fd044f0b919629933ea56e9c from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59219)
I20240501 14:09:33.765873 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 66436017fd044f0b919629933ea56e9c from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59219: session id = 8a13c028ba8a4a659b74af6c265df95e-66436017fd044f0b919629933ea56e9c
I20240501 14:09:33.767369 12607 tablet_copy_source_session.cc:205] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.769202 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66436017fd044f0b919629933ea56e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.769500 15849 ts_tablet_manager.cc:902] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:33.770273 15849 tablet_copy_client.cc:308] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:33.771562 15833 tablet_copy_client.cc:783] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.771736 15833 tablet_copy_client.cc:647] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.772254 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d5f20c67fc48aeacc0c9b0a9b26a55" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:33.772282 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36d5f20c67fc48aeacc0c9b0a9b26a55" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:33.772405 12717 raft_consensus.cc:2454] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:33.773406 15833 tablet_copy_client.cc:523] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.775537 15833 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:33.775642 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea946f39526401eb9b684f60cd9e8d0" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:33.775769 15353 raft_consensus.cc:2454] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:33.775982 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea946f39526401eb9b684f60cd9e8d0" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:33.776088 15352 raft_consensus.cc:3037] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.777156 15352 raft_consensus.cc:2454] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:33.783071 12980 consensus_queue.cc:237] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.784667 12717 raft_consensus.cc:1261] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.785382 12586 raft_consensus.cc:1261] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.785604 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet b547fcb34f12402abccdb23678c95e26 from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:34877)
I20240501 14:09:33.785691 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet b547fcb34f12402abccdb23678c95e26 from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:34877: session id = f23f426391724d6688aafffe7ee017d8-b547fcb34f12402abccdb23678c95e26
I20240501 14:09:33.786605 12607 tablet_copy_source_session.cc:205] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.787074 15684 consensus_queue.cc:1035] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.787953 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b547fcb34f12402abccdb23678c95e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.789891 15849 tablet_copy_client.cc:783] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.790048 15849 tablet_copy_client.cc:647] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.791859 15684 consensus_queue.cc:1035] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:33.792464 12915 consensus_peers.cc:485] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ea1af845a8e4f39a973d1f8ef691762. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.792587 15684 raft_consensus.cc:2932] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.793956 15849 tablet_copy_client.cc:523] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.795243 15849 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:33.795151 12717 raft_consensus.cc:2932] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.795651 12437 catalog_manager.cc:5484] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.797250 12586 raft_consensus.cc:2932] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.802724 15863 ts_tablet_manager.cc:902] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:33.803567 15863 tablet_copy_client.cc:308] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:33.805260 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 6305f3f7cb8846bf843d2d753fd1ae07 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58523)
I20240501 14:09:33.805382 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 6305f3f7cb8846bf843d2d753fd1ae07 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58523: session id = ab90e77f0027449bb3155980ca95593c-6305f3f7cb8846bf843d2d753fd1ae07
I20240501 14:09:33.806483 12738 tablet_copy_source_session.cc:205] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.808135 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6305f3f7cb8846bf843d2d753fd1ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.810495 15863 tablet_copy_client.cc:783] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.810648 15863 tablet_copy_client.cc:647] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.812525 15863 tablet_copy_client.cc:523] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.813180 15833 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.813480 15833 tablet_bootstrap.cc:492] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:33.813637 15833 ts_tablet_manager.cc:1367] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.038s user 0.029s sys 0.000s
I20240501 14:09:33.814049 15833 raft_consensus.cc:348] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.814129 15863 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:33.814173 15833 raft_consensus.cc:729] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:33.814266 15833 consensus_queue.cc:260] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.815027 15833 ts_tablet_manager.cc:1398] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:33.815615 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-66436017fd044f0b919629933ea56e9c received from {username='slave'} at 127.6.82.5:59219
I20240501 14:09:33.815698 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-66436017fd044f0b919629933ea56e9c on tablet 66436017fd044f0b919629933ea56e9c with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:33.823276 15819 raft_consensus.cc:482] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:33.823390 15819 raft_consensus.cc:504] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.823779 15819 leader_election.cc:288] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:33.824182 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab31238e6b4ea4b0da2561c79e7efd" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:33.824301 15352 raft_consensus.cc:2454] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:33.824775 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab31238e6b4ea4b0da2561c79e7efd" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:33.824888 14958 raft_consensus.cc:2454] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:33.825322 15157 leader_election.cc:302] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:33.827610 12849 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.830099 12717 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.830585 12980 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.830637 15551 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.831455 15819 raft_consensus.cc:2781] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:33.831516 15819 raft_consensus.cc:482] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:33.831552 15819 raft_consensus.cc:3037] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.832541 15819 raft_consensus.cc:504] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.832902 15819 leader_election.cc:288] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:33.833281 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab31238e6b4ea4b0da2561c79e7efd" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:33.833298 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab31238e6b4ea4b0da2561c79e7efd" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:33.833391 15352 raft_consensus.cc:3037] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.833410 14958 raft_consensus.cc:3037] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.834270 15352 raft_consensus.cc:2454] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:33.834370 14958 raft_consensus.cc:2454] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:33.834637 15157 leader_election.cc:302] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:33.834991 15551 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.836894 15819 raft_consensus.cc:2781] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:33.836977 15819 raft_consensus.cc:686] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:33.837105 15819 consensus_queue.cc:237] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:33.838605 15551 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.838972 12437 catalog_manager.cc:5484] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.840646 15849 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.841643 12438 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.843277 12978 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.843422 15849 tablet_bootstrap.cc:492] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:33.844238 15849 ts_tablet_manager.cc:1367] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.049s user 0.031s sys 0.000s
I20240501 14:09:33.844746 15849 raft_consensus.cc:348] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.847607 15833 ts_tablet_manager.cc:902] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:33.848053 15833 tablet_copy_client.cc:308] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:33.848737 15863 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.849104 15863 tablet_bootstrap.cc:492] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:33.849315 15863 ts_tablet_manager.cc:1367] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.000s
I20240501 14:09:33.849572 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet d178c3291efc44369e3adb921ccc6517 from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59219)
I20240501 14:09:33.849678 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet d178c3291efc44369e3adb921ccc6517 from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59219: session id = 8a13c028ba8a4a659b74af6c265df95e-d178c3291efc44369e3adb921ccc6517
I20240501 14:09:33.849795 15863 raft_consensus.cc:348] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.849910 15863 raft_consensus.cc:729] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:33.850008 15863 consensus_queue.cc:260] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.850312 15863 ts_tablet_manager.cc:1398] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:33.850603 12607 tablet_copy_source_session.cc:205] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.850908 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-6305f3f7cb8846bf843d2d753fd1ae07 received from {username='slave'} at 127.6.82.6:58523
I20240501 14:09:33.850992 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session ab90e77f0027449bb3155980ca95593c-6305f3f7cb8846bf843d2d753fd1ae07 on tablet 6305f3f7cb8846bf843d2d753fd1ae07 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:33.851491 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d178c3291efc44369e3adb921ccc6517. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:09:33.853116 12785 consensus_peers.cc:485] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d64234effb3446bba1ff82bdf37acaa7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.853351 15833 tablet_copy_client.cc:783] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.844873 15849 raft_consensus.cc:729] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:33.853479 15833 tablet_copy_client.cc:647] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.853549 15849 consensus_queue.cc:260] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.853871 15849 ts_tablet_manager.cc:1398] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:33.854861 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-b547fcb34f12402abccdb23678c95e26 received from {username='slave'} at 127.6.82.8:34877
I20240501 14:09:33.854933 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session f23f426391724d6688aafffe7ee017d8-b547fcb34f12402abccdb23678c95e26 on tablet b547fcb34f12402abccdb23678c95e26 with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:33.855755 12717 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.857692 15833 tablet_copy_client.cc:523] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.858868 15833 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:33.865096 14958 consensus_queue.cc:237] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:33.866762 15221 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.867401 15573 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:33.869886 15573 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
W20240501 14:09:33.871420 14893 consensus_peers.cc:485] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dd29fa5dc2b4408878ceb3c12a57098. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.871484 12849 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.871697 15221 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:33.872129 15421 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:33.872442 12437 catalog_manager.cc:5484] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.873133 12849 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:33.874699 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:33.879302 15221 raft_consensus.cc:1261] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.880546 15425 consensus_queue.cc:1035] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.882632 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (table rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63]). Expected version 1 got 0
I20240501 14:09:33.882695 12717 raft_consensus.cc:3037] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:33.883823 12717 raft_consensus.cc:1261] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:33.883888 15863 ts_tablet_manager.cc:902] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:33.884300 15863 tablet_copy_client.cc:308] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:33.884296 15425 consensus_queue.cc:1035] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:33.885008 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet afa5ebe09c99435fb69ceb527ab4734a from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58029)
I20240501 14:09:33.885134 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet afa5ebe09c99435fb69ceb527ab4734a from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58029: session id = ab90e77f0027449bb3155980ca95593c-afa5ebe09c99435fb69ceb527ab4734a
I20240501 14:09:33.885959 12607 tablet_copy_source_session.cc:205] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.887493 15470 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.887826 15470 tablet.cc:1620] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:33.887475 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5ebe09c99435fb69ceb527ab4734a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.889007 15647 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.889309 15647 tablet.cc:1620] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:33.889283 12438 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (table rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63]). Expected version 1 got 0
I20240501 14:09:33.890350 15863 tablet_copy_client.cc:783] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.890491 15863 tablet_copy_client.cc:647] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.891646 15833 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.892009 15863 tablet_copy_client.cc:523] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.892204 15833 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:33.892364 15833 ts_tablet_manager.cc:1367] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.000s
I20240501 14:09:33.892997 12437 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63] alter complete (version 1)
I20240501 14:09:33.893214 15863 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:33.893560 15833 raft_consensus.cc:348] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.893682 15833 raft_consensus.cc:729] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:33.893774 15833 consensus_queue.cc:260] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.894819 15833 ts_tablet_manager.cc:1398] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.895512 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-d178c3291efc44369e3adb921ccc6517 received from {username='slave'} at 127.6.82.5:59219
I20240501 14:09:33.895581 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-d178c3291efc44369e3adb921ccc6517 on tablet d178c3291efc44369e3adb921ccc6517 with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:33.896423 15907 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:33.898022 12439 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (table rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63]). Expected version 1 got 0
I20240501 14:09:33.899034 15907 tablet.cc:1620] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:33.899799 15909 ts_tablet_manager.cc:902] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:33.900578 15909 tablet_copy_client.cc:308] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:33.905728 15833 ts_tablet_manager.cc:902] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:33.905871 12978 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.906139 15833 tablet_copy_client.cc:308] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:33.907367 12718 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.907495 12849 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:33.907766 12914 consensus_peers.cc:485] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b5c40793b348bbb0e5251c8d553e30. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.907876 14674 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.908123 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:09:33.908170 14674 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.908236 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 14:09:33.911438 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 7693e3e204634676a758726bb6d3d2ce from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:51545)
I20240501 14:09:33.911548 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 7693e3e204634676a758726bb6d3d2ce from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:51545: session id = 66e2dc34330848c3b8c2dd15898d34c8-7693e3e204634676a758726bb6d3d2ce
I20240501 14:09:33.912032 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f
I20240501 14:09:33.912374 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet 2b152ca89c5a416f9b1a65217e4a950e
I20240501 14:09:33.912533 12607 tablet_copy_source_session.cc:205] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.912705 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 14f1774f12d34e91a9a5409a60ee1976
I20240501 14:09:33.913096 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet abac52c109174021ac11360b0d22ccfa from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:37435)
I20240501 14:09:33.913187 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet abac52c109174021ac11360b0d22ccfa from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:37435: session id = 8a13c028ba8a4a659b74af6c265df95e-abac52c109174021ac11360b0d22ccfa
I20240501 14:09:33.913336 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:33.913612 15922 tablet_replica.cc:330] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:33.914047 12738 tablet_copy_source_session.cc:205] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.914218 15909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693e3e204634676a758726bb6d3d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.914330 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:33.914474 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:33.914691 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f1774f12d34e91a9a5409a60ee1976 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:33.915488 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:33.915526 15922 raft_consensus.cc:2229] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.915695 15922 raft_consensus.cc:2258] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.916345 15922 ts_tablet_manager.cc:1875] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.916452 15909 tablet_copy_client.cc:783] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
W20240501 14:09:33.916525 12432 catalog_manager.cc:4797] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): delete failed for tablet 2b152ca89c5a416f9b1a65217e4a950e because the tablet was not found. No further retry: Not found: Tablet not found: 2b152ca89c5a416f9b1a65217e4a950e
I20240501 14:09:33.916622 15909 tablet_copy_client.cc:647] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.917173 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abac52c109174021ac11360b0d22ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.917161 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:33.917188 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f1774f12d34e91a9a5409a60ee1976 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:33.919085 15923 tablet_replica.cc:330] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:33.919389 15833 tablet_copy_client.cc:783] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.919418 15923 raft_consensus.cc:2229] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.919531 15833 tablet_copy_client.cc:647] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.919595 15923 raft_consensus.cc:2258] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.920432 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:33.920746 15909 tablet_copy_client.cc:523] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.920948 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 14f1774f12d34e91a9a5409a60ee1976 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:33.921252 15925 tablet_replica.cc:330] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:33.921388 15925 raft_consensus.cc:2229] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.921442 15833 tablet_copy_client.cc:523] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.921577 15925 raft_consensus.cc:2258] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.922022 15925 ts_tablet_manager.cc:1875] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.922173 15909 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.922482 15833 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:33.924139 15925 ts_tablet_manager.cc:1888] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.924207 15925 log.cc:1196] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/14f1774f12d34e91a9a5409a60ee1976
I20240501 14:09:33.924526 15925 ts_tablet_manager.cc:1909] T 14f1774f12d34e91a9a5409a60ee1976 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:33.925122 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 14f1774f12d34e91a9a5409a60ee1976 (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.920055 15923 ts_tablet_manager.cc:1875] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.925480 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b152ca89c5a416f9b1a65217e4a950e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:33 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:33.926213 15925 tablet_replica.cc:330] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:33.926388 15925 raft_consensus.cc:2229] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.926575 15924 tablet_replica.cc:330] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:33.926775 15924 raft_consensus.cc:2229] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.926944 15924 raft_consensus.cc:2258] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.926810 15925 raft_consensus.cc:2258] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.927459 15924 ts_tablet_manager.cc:1875] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.927670 15922 ts_tablet_manager.cc:1888] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.927790 15922 log.cc:1196] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/2b152ca89c5a416f9b1a65217e4a950e
I20240501 14:09:33.927987 15925 ts_tablet_manager.cc:1875] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.928100 15922 ts_tablet_manager.cc:1909] T 2b152ca89c5a416f9b1a65217e4a950e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:33.928798 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 2b152ca89c5a416f9b1a65217e4a950e (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.929028 15922 tablet_replica.cc:330] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:33.929186 15922 raft_consensus.cc:2229] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.929358 15922 raft_consensus.cc:2258] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.928966 15923 ts_tablet_manager.cc:1888] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.929611 15923 log.cc:1196] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/bdd40853ff8a42c4b5eaa3fa57bd0c3f
I20240501 14:09:33.929764 15925 ts_tablet_manager.cc:1888] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.929841 15925 log.cc:1196] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/2b152ca89c5a416f9b1a65217e4a950e
I20240501 14:09:33.929890 15923 ts_tablet_manager.cc:1909] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:33.930068 15922 ts_tablet_manager.cc:1875] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.930084 15925 ts_tablet_manager.cc:1909] T 2b152ca89c5a416f9b1a65217e4a950e P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:33.930650 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.930972 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 2b152ca89c5a416f9b1a65217e4a950e (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.931839 15924 ts_tablet_manager.cc:1888] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.931931 15924 log.cc:1196] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/bdd40853ff8a42c4b5eaa3fa57bd0c3f
I20240501 14:09:33.932215 15924 ts_tablet_manager.cc:1909] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:33.932790 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.932854 15923 tablet_replica.cc:330] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:33.932987 15923 raft_consensus.cc:2229] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:33.933243 15923 raft_consensus.cc:2258] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.933442 15924 tablet_replica.cc:330] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:33.933553 15924 raft_consensus.cc:2229] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.933650 15924 raft_consensus.cc:2258] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.933766 15922 ts_tablet_manager.cc:1888] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.933869 15922 log.cc:1196] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdd40853ff8a42c4b5eaa3fa57bd0c3f
I20240501 14:09:33.934095 15924 ts_tablet_manager.cc:1875] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.934259 15922 ts_tablet_manager.cc:1909] T bdd40853ff8a42c4b5eaa3fa57bd0c3f P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:33.934736 15923 ts_tablet_manager.cc:1875] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.934983 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet bdd40853ff8a42c4b5eaa3fa57bd0c3f (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.935199 15922 tablet_replica.cc:330] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:33.935407 15922 raft_consensus.cc:2229] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:33.935586 15922 raft_consensus.cc:2258] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:33.936035 15922 ts_tablet_manager.cc:1875] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:33.937803 15922 ts_tablet_manager.cc:1888] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.937924 15922 log.cc:1196] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/14f1774f12d34e91a9a5409a60ee1976
I20240501 14:09:33.937920 15923 ts_tablet_manager.cc:1888] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:33.938004 15923 log.cc:1196] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b152ca89c5a416f9b1a65217e4a950e
I20240501 14:09:33.938300 15922 ts_tablet_manager.cc:1909] T 14f1774f12d34e91a9a5409a60ee1976 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:33.938977 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 14f1774f12d34e91a9a5409a60ee1976 (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.939183 15924 ts_tablet_manager.cc:1888] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:33.939412 15923 ts_tablet_manager.cc:1909] T 2b152ca89c5a416f9b1a65217e4a950e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:33.940196 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 2b152ca89c5a416f9b1a65217e4a950e (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.940506 15924 log.cc:1196] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/14f1774f12d34e91a9a5409a60ee1976
I20240501 14:09:33.940830 15924 ts_tablet_manager.cc:1909] T 14f1774f12d34e91a9a5409a60ee1976 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:33.941546 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 14f1774f12d34e91a9a5409a60ee1976 (table rebalance_test_table_4 [id=754b0d8e580e4f4397b4637bda294224]) successfully deleted
I20240501 14:09:33.943974 14674 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.945847 12849 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.948958 12439 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.954288 15833 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.954793 15833 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:33.954975 15833 ts_tablet_manager.cc:1367] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.000s
I20240501 14:09:33.954947 12718 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:33.955484 15833 raft_consensus.cc:348] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.955582 15833 raft_consensus.cc:729] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:33.955667 15833 consensus_queue.cc:260] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:33.955920 15833 ts_tablet_manager.cc:1398] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.956427 15863 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.956681 15863 tablet_bootstrap.cc:492] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:33.957617 15932 ts_tablet_manager.cc:902] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:33.957690 15863 ts_tablet_manager.cc:1367] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.065s user 0.029s sys 0.000s
I20240501 14:09:33.958168 15932 tablet_copy_client.cc:308] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:33.958163 15863 raft_consensus.cc:348] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.958266 15863 raft_consensus.cc:729] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:33.958370 15863 consensus_queue.cc:260] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.958683 15863 ts_tablet_manager.cc:1398] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:33.958701 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-abac52c109174021ac11360b0d22ccfa received from {username='slave'} at 127.6.82.5:37435
I20240501 14:09:33.958812 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-abac52c109174021ac11360b0d22ccfa on tablet abac52c109174021ac11360b0d22ccfa with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:33.960474 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-afa5ebe09c99435fb69ceb527ab4734a received from {username='slave'} at 127.6.82.6:58029
I20240501 14:09:33.960551 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session ab90e77f0027449bb3155980ca95593c-afa5ebe09c99435fb69ceb527ab4734a on tablet afa5ebe09c99435fb69ceb527ab4734a with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:33.961968 15909 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:33.962265 15909 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:33.962424 15909 ts_tablet_manager.cc:1367] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.004s
I20240501 14:09:33.963805 15909 raft_consensus.cc:348] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:33.963948 15909 raft_consensus.cc:729] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:33.964072 15909 consensus_queue.cc:260] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:33.964361 15909 ts_tablet_manager.cc:1398] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:33.964812 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-7693e3e204634676a758726bb6d3d2ce received from {username='slave'} at 127.6.82.7:51545
I20240501 14:09:33.964896 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-7693e3e204634676a758726bb6d3d2ce on tablet 7693e3e204634676a758726bb6d3d2ce with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:33.964990 13000 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 8147fae0de98429ab6554c7e7f5bf8ee from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:34873)
I20240501 14:09:33.965118 13000 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 8147fae0de98429ab6554c7e7f5bf8ee from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:34873: session id = 66e2dc34330848c3b8c2dd15898d34c8-8147fae0de98429ab6554c7e7f5bf8ee
I20240501 14:09:33.966430 13000 tablet_copy_source_session.cc:205] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.967644 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8147fae0de98429ab6554c7e7f5bf8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.968047 12978 consensus_queue.cc:237] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:33.969604 15932 tablet_copy_client.cc:783] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.969743 15932 tablet_copy_client.cc:647] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.969789 12585 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:33.969875 12915 consensus_peers.cc:485] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2631324f66d454697b895ecdd227276. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.970173 15683 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.970535 12718 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.970856 15683 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.971225 15932 tablet_copy_client.cc:523] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.972339 15932 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:33.972321 14674 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.973691 12585 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.975373 12438 catalog_manager.cc:5484] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:33.976071 12718 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:33.986012 12849 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:33.987087 15849 ts_tablet_manager.cc:902] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:33.987563 15849 tablet_copy_client.cc:308] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
W20240501 14:09:33.987943 12785 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd5489eab5b04751bc8b3e5fb9e8cf73. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:33.988087 12978 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.988229 12718 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:33.988591 15551 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.988818 15879 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:33.990669 15879 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.992064 13000 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 80afbd22fdad4bcdad3768c4a1953f0d from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:45593)
I20240501 14:09:33.992152 13000 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 80afbd22fdad4bcdad3768c4a1953f0d from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:45593: session id = f23f426391724d6688aafffe7ee017d8-80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:33.992882 13000 tablet_copy_source_session.cc:205] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:33.993721 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80afbd22fdad4bcdad3768c4a1953f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:33.993714 12978 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.995693 15849 tablet_copy_client.cc:783] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:33.995853 15849 tablet_copy_client.cc:647] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:33.996928 12439 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:33.997941 15849 tablet_copy_client.cc:523] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:33.999264 15849 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:33.999158 12718 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:34.000439 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:34.001204 15932 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.001613 15932 tablet_bootstrap.cc:492] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:34.001924 15932 ts_tablet_manager.cc:1367] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20240501 14:09:34.002444 15932 raft_consensus.cc:348] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:34.002610 15932 raft_consensus.cc:729] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:34.002846 15932 consensus_queue.cc:260] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:34.003723 15932 ts_tablet_manager.cc:1398] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.004356 13000 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-8147fae0de98429ab6554c7e7f5bf8ee received from {username='slave'} at 127.6.82.7:34873
I20240501 14:09:34.004446 13000 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-8147fae0de98429ab6554c7e7f5bf8ee on tablet 8147fae0de98429ab6554c7e7f5bf8ee with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:34.010600 15470 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.010950 15470 tablet.cc:1620] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:34.010978 15647 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.011226 15647 tablet.cc:1620] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:34.014887 15907 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.015048 15932 ts_tablet_manager.cc:902] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:34.015164 15907 tablet.cc:1620] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:34.015455 15932 tablet_copy_client.cc:308] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:34.016315 12444 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=0bddc1246627417e96e533c99426ee63] alter complete (version 2)
I20240501 14:09:34.017050 13000 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 49f2b0e67d3c43d584ac7583b98caeb9 from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:34873)
I20240501 14:09:34.017170 15833 ts_tablet_manager.cc:902] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:34.017172 13000 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 49f2b0e67d3c43d584ac7583b98caeb9 from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:34873: session id = 66e2dc34330848c3b8c2dd15898d34c8-49f2b0e67d3c43d584ac7583b98caeb9
I20240501 14:09:34.017606 15833 tablet_copy_client.cc:308] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:34.018144 13000 tablet_copy_source_session.cc:205] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.018222 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 63a76e89d58d48ba9d1b3b98ef6c172d from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59069)
I20240501 14:09:34.018319 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 63a76e89d58d48ba9d1b3b98ef6c172d from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59069: session id = 8a13c028ba8a4a659b74af6c265df95e-63a76e89d58d48ba9d1b3b98ef6c172d
I20240501 14:09:34.019075 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f2b0e67d3c43d584ac7583b98caeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.019241 12999 tablet_copy_source_session.cc:205] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.020884 15932 tablet_copy_client.cc:783] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.021019 15932 tablet_copy_client.cc:647] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.022519 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a76e89d58d48ba9d1b3b98ef6c172d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.024514 15833 tablet_copy_client.cc:783] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.024657 15833 tablet_copy_client.cc:647] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.027238 15932 tablet_copy_client.cc:523] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.028172 15932 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:34.031723 15833 tablet_copy_client.cc:523] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.032967 15833 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:34.041764 15849 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.042109 15849 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:34.042309 15849 ts_tablet_manager.cc:1367] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.043s user 0.028s sys 0.000s
I20240501 14:09:34.042781 15849 raft_consensus.cc:348] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:34.042966 15849 raft_consensus.cc:729] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:34.043118 15849 consensus_queue.cc:260] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:34.043977 15849 ts_tablet_manager.cc:1398] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.044503 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-80afbd22fdad4bcdad3768c4a1953f0d received from {username='slave'} at 127.6.82.8:45593
I20240501 14:09:34.044596 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session f23f426391724d6688aafffe7ee017d8-80afbd22fdad4bcdad3768c4a1953f0d on tablet 80afbd22fdad4bcdad3768c4a1953f0d with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:34.052297 15932 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.053061 15932 tablet_bootstrap.cc:492] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:34.053243 15932 ts_tablet_manager.cc:1367] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240501 14:09:34.053675 15932 raft_consensus.cc:348] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:34.053800 15932 raft_consensus.cc:729] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:34.053901 15932 consensus_queue.cc:260] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:34.054265 15932 ts_tablet_manager.cc:1398] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:34.054703 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-49f2b0e67d3c43d584ac7583b98caeb9 received from {username='slave'} at 127.6.82.7:34873
I20240501 14:09:34.054800 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-49f2b0e67d3c43d584ac7583b98caeb9 on tablet 49f2b0e67d3c43d584ac7583b98caeb9 with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:34.072772 15833 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.073863 15833 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:34.074041 15833 ts_tablet_manager.cc:1367] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.000s
I20240501 14:09:34.074519 15833 raft_consensus.cc:348] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:34.074630 15833 raft_consensus.cc:729] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:34.074708 15833 consensus_queue.cc:260] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:34.075721 15833 ts_tablet_manager.cc:1398] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.076210 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-63a76e89d58d48ba9d1b3b98ef6c172d received from {username='slave'} at 127.6.82.5:59069
I20240501 14:09:34.076305 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-63a76e89d58d48ba9d1b3b98ef6c172d on tablet 63a76e89d58d48ba9d1b3b98ef6c172d with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:34.102864 15863 ts_tablet_manager.cc:902] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:34.103358 15863 tablet_copy_client.cc:308] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:34.103860 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet e13e55ae52a949bc893de94082130157 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58523)
I20240501 14:09:34.103979 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet e13e55ae52a949bc893de94082130157 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58523: session id = ab90e77f0027449bb3155980ca95593c-e13e55ae52a949bc893de94082130157
I20240501 14:09:34.104843 12738 tablet_copy_source_session.cc:205] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.105901 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e13e55ae52a949bc893de94082130157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.107765 15863 tablet_copy_client.cc:783] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.107914 15863 tablet_copy_client.cc:647] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.109265 15863 tablet_copy_client.cc:523] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.110491 15863 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.123329 12444 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:34.128548 15470 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.129644 15907 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.130479 15647 tablet.cc:1722] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.135836 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=0bddc1246627417e96e533c99426ee63] alter complete (version 3)
I20240501 14:09:34.140543 15863 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.140890 15863 tablet_bootstrap.cc:492] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:34.141081 15863 ts_tablet_manager.cc:1367] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.000s
I20240501 14:09:34.141561 15863 raft_consensus.cc:348] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:34.141680 15863 raft_consensus.cc:729] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:34.141777 15863 consensus_queue.cc:260] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:34.142050 15863 ts_tablet_manager.cc:1398] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:34.142697 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-e13e55ae52a949bc893de94082130157 received from {username='slave'} at 127.6.82.6:58523
I20240501 14:09:34.142799 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session ab90e77f0027449bb3155980ca95593c-e13e55ae52a949bc893de94082130157 on tablet e13e55ae52a949bc893de94082130157 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:34.145684 15352 raft_consensus.cc:1203] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.151896 15573 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.154980 15573 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.199553 14958 raft_consensus.cc:1203] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.231921 15819 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.238863 15572 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.239459 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.239584 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.242058 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 36d5f20c67fc48aeacc0c9b0a9b26a55
I20240501 14:09:34.242657 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:34.243532 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:34.243769 15070 tablet_service.cc:1514] Processing DeleteTablet for tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:34.243808 15923 tablet_replica.cc:330] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:34.243973 15923 raft_consensus.cc:2229] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.244127 15923 raft_consensus.cc:2258] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.244946 15923 ts_tablet_manager.cc:1875] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.247051 16105 tablet_replica.cc:330] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:34.247151 15923 ts_tablet_manager.cc:1888] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.247187 16105 raft_consensus.cc:2229] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.247215 15923 log.cc:1196] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/36d5f20c67fc48aeacc0c9b0a9b26a55
I20240501 14:09:34.247304 16105 raft_consensus.cc:2258] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.248771 15923 ts_tablet_manager.cc:1909] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:34.249177 16105 ts_tablet_manager.cc:1875] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.250483 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (table rebalancer_dynamic_table_new_name [id=0bddc1246627417e96e533c99426ee63]) successfully deleted
I20240501 14:09:34.253072 16105 ts_tablet_manager.cc:1888] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.253156 16105 log.cc:1196] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/36d5f20c67fc48aeacc0c9b0a9b26a55
I20240501 14:09:34.253439 16105 ts_tablet_manager.cc:1909] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:34.254076 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (table rebalancer_dynamic_table_new_name [id=0bddc1246627417e96e533c99426ee63]) successfully deleted
I20240501 14:09:34.256060 16106 tablet_replica.cc:330] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:34.256201 16106 raft_consensus.cc:2229] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.256397 16106 raft_consensus.cc:2258] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.256999 16106 ts_tablet_manager.cc:1875] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.259588 16106 ts_tablet_manager.cc:1888] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.259717 16106 log.cc:1196] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/36d5f20c67fc48aeacc0c9b0a9b26a55
I20240501 14:09:34.260071 16106 ts_tablet_manager.cc:1909] T 36d5f20c67fc48aeacc0c9b0a9b26a55 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:34.260775 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 36d5f20c67fc48aeacc0c9b0a9b26a55 (table rebalancer_dynamic_table_new_name [id=0bddc1246627417e96e533c99426ee63]) successfully deleted
I20240501 14:09:34.269109 15089 raft_consensus.cc:1203] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.283629 15200 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.286034 12566 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:34.292126 14938 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:34.295037 15863 ts_tablet_manager.cc:902] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:34.295482 15863 tablet_copy_client.cc:308] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:34.297318 15332 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.299523 15849 ts_tablet_manager.cc:902] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:34.302306 15849 tablet_copy_client.cc:308] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:34.304190 12698 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:34.306196 15070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:09:34.308646 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet d64234effb3446bba1ff82bdf37acaa7 from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:58949)
I20240501 14:09:34.308780 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet d64234effb3446bba1ff82bdf37acaa7 from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:58949: session id = f23f426391724d6688aafffe7ee017d8-d64234effb3446bba1ff82bdf37acaa7
I20240501 14:09:34.310194 12869 tablet_copy_source_session.cc:205] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.311568 15849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64234effb3446bba1ff82bdf37acaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.313824 15849 tablet_copy_client.cc:783] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.313971 15849 tablet_copy_client.cc:647] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.317584 12829 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:34.318790 12960 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:34.319931 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 1ea1af845a8e4f39a973d1f8ef691762 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:42689)
I20240501 14:09:34.320016 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 1ea1af845a8e4f39a973d1f8ef691762 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:42689: session id = ab90e77f0027449bb3155980ca95593c-1ea1af845a8e4f39a973d1f8ef691762
I20240501 14:09:34.320904 12999 tablet_copy_source_session.cc:205] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.322974 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea1af845a8e4f39a973d1f8ef691762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.324977 15863 tablet_copy_client.cc:783] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.325103 15863 tablet_copy_client.cc:647] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.327304 15863 tablet_copy_client.cc:523] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.328475 15863 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.341269 15849 tablet_copy_client.cc:523] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.342873 15849 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:34.348486 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:34.353818 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0653b46a484e74b3ee0b670ade07cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc]), partition=
I20240501 14:09:34.354069 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0653b46a484e74b3ee0b670ade07cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.354102 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0653b46a484e74b3ee0b670ade07cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc]), partition=
I20240501 14:09:34.354162 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0653b46a484e74b3ee0b670ade07cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc]), partition=
I20240501 14:09:34.354363 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0653b46a484e74b3ee0b670ade07cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.354393 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0653b46a484e74b3ee0b670ade07cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.358335 16133 tablet_bootstrap.cc:492] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:34.358647 16131 tablet_bootstrap.cc:492] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.360476 16132 tablet_bootstrap.cc:492] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:34.361269 16132 tablet_bootstrap.cc:654] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.361989 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:34.370275 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 0c41acbf912c4dc0a68fe5b99df2f84b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.370573 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41acbf912c4dc0a68fe5b99df2f84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.372440 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 0c41acbf912c4dc0a68fe5b99df2f84b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.372733 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41acbf912c4dc0a68fe5b99df2f84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.374274 12566 tablet_service.cc:1467] Processing CreateTablet for tablet f4010a078895473980e7108269ea7bc1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.374536 16133 tablet_bootstrap.cc:654] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.374581 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4010a078895473980e7108269ea7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.366393 15573 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.373675 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 188ce319682147ddaad30aae7232a311 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.375366 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188ce319682147ddaad30aae7232a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.375553 16131 tablet_bootstrap.cc:654] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.376672 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 188ce319682147ddaad30aae7232a311 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.376906 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188ce319682147ddaad30aae7232a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.376981 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 0c41acbf912c4dc0a68fe5b99df2f84b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.377187 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41acbf912c4dc0a68fe5b99df2f84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.377488 15070 tablet_service.cc:1467] Processing CreateTablet for tablet f4010a078895473980e7108269ea7bc1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.377684 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4010a078895473980e7108269ea7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.374642 14938 tablet_service.cc:1467] Processing CreateTablet for tablet f4010a078895473980e7108269ea7bc1 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.379379 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 188ce319682147ddaad30aae7232a311 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:34.379557 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4010a078895473980e7108269ea7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.379648 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188ce319682147ddaad30aae7232a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.380162 15833 ts_tablet_manager.cc:902] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:34.380592 15833 tablet_copy_client.cc:308] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:34.381615 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 25b5c40793b348bbb0e5251c8d553e30 from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59069)
I20240501 14:09:34.381711 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 25b5c40793b348bbb0e5251c8d553e30 from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59069: session id = 8a13c028ba8a4a659b74af6c265df95e-25b5c40793b348bbb0e5251c8d553e30
I20240501 14:09:34.382545 12999 tablet_copy_source_session.cc:205] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.384565 16133 tablet_bootstrap.cc:492] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:34.384660 16133 ts_tablet_manager.cc:1367] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20240501 14:09:34.385696 16133 raft_consensus.cc:348] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.385905 16133 raft_consensus.cc:374] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.385905 15573 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.385993 16133 raft_consensus.cc:729] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.386138 16133 consensus_queue.cc:260] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.386482 16133 ts_tablet_manager.cc:1398] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:34.387246 16131 tablet_bootstrap.cc:492] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:34.387327 16131 ts_tablet_manager.cc:1367] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 14:09:34.387727 16131 raft_consensus.cc:348] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.387825 16131 raft_consensus.cc:374] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.387851 16131 raft_consensus.cc:729] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.387981 16131 consensus_queue.cc:260] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.388921 16131 ts_tablet_manager.cc:1398] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.389072 16131 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.389914 16131 tablet_bootstrap.cc:654] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.391883 15222 raft_consensus.cc:1203] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.392611 16138 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:34.394318 16138 tablet_bootstrap.cc:654] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.396581 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25b5c40793b348bbb0e5251c8d553e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.397215 16131 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:34.397294 16131 ts_tablet_manager.cc:1367] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:34.397727 16131 raft_consensus.cc:348] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.397853 16131 raft_consensus.cc:374] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.397888 16131 raft_consensus.cc:729] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.398128 16131 consensus_queue.cc:260] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.398505 16131 ts_tablet_manager.cc:1398] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.398784 16131 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.400839 16131 tablet_bootstrap.cc:654] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.401479 16143 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:34.403008 16143 tablet_bootstrap.cc:654] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.408042 15221 raft_consensus.cc:1203] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.408617 16131 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:34.408707 16131 ts_tablet_manager.cc:1367] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:34.409166 16131 raft_consensus.cc:348] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.409278 16131 raft_consensus.cc:374] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.409312 16131 raft_consensus.cc:729] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.409418 16131 consensus_queue.cc:260] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.409683 16131 ts_tablet_manager.cc:1398] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.409852 16131 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.410176 16143 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:34.410252 16143 ts_tablet_manager.cc:1367] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:34.410624 16143 raft_consensus.cc:348] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.410714 16143 raft_consensus.cc:374] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.410820 16143 raft_consensus.cc:729] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.410926 16143 consensus_queue.cc:260] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.411180 16143 ts_tablet_manager.cc:1398] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.411716 15863 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.411847 16131 tablet_bootstrap.cc:654] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.411970 15863 tablet_bootstrap.cc:492] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:34.412135 15863 ts_tablet_manager.cc:1367] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.084s user 0.026s sys 0.000s
I20240501 14:09:34.412482 15863 raft_consensus.cc:348] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:34.412590 15863 raft_consensus.cc:729] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:34.412675 15863 consensus_queue.cc:260] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:34.413376 16138 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:34.413514 16138 ts_tablet_manager.cc:1367] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:34.413942 16138 raft_consensus.cc:348] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.414103 16138 raft_consensus.cc:374] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.414176 16138 raft_consensus.cc:729] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.414371 16138 consensus_queue.cc:260] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.414491 16150 ts_tablet_manager.cc:902] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:34.414827 16138 ts_tablet_manager.cc:1398] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.415040 16138 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:34.417376 12442 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:09:34.417496 12442 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 14:09:34.417708 15089 raft_consensus.cc:1203] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.418547 16150 tablet_copy_client.cc:308] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:34.419061 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:58949)
I20240501 14:09:34.419173 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet bd5489eab5b04751bc8b3e5fb9e8cf73 from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:58949: session id = f23f426391724d6688aafffe7ee017d8-bd5489eab5b04751bc8b3e5fb9e8cf73
I20240501 14:09:34.419186 16131 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:34.419266 16131 ts_tablet_manager.cc:1367] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:34.419981 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet 1ea1af845a8e4f39a973d1f8ef691762
I20240501 14:09:34.420161 12869 tablet_copy_source_session.cc:205] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.420316 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 5e22abf2e2e544eb9ac8dccca71117bc
I20240501 14:09:34.420579 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet c7fc40ff90834e84a781cd08a45a33f8
I20240501 14:09:34.420888 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e22abf2e2e544eb9ac8dccca71117bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:34.420884 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:34.421042 15924 tablet_replica.cc:330] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:34.421058 15923 tablet_replica.cc:330] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:34.421161 15924 raft_consensus.cc:2229] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.421197 15923 raft_consensus.cc:2229] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.421300 15924 raft_consensus.cc:2258] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.421350 15923 raft_consensus.cc:2258] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.421684 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:34.421789 15924 ts_tablet_manager.cc:1875] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.421808 16138 tablet_bootstrap.cc:654] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.422117 15923 ts_tablet_manager.cc:1875] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:09:34.422230 12431 catalog_manager.cc:4810] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): delete failed for tablet 1ea1af845a8e4f39a973d1f8ef691762 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea1af845a8e4f39a973d1f8ef691762 already in progress: copying tablet
I20240501 14:09:34.423960 15924 ts_tablet_manager.cc:1888] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.424026 15924 log.cc:1196] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/5e22abf2e2e544eb9ac8dccca71117bc
I20240501 14:09:34.424314 15924 ts_tablet_manager.cc:1909] T 5e22abf2e2e544eb9ac8dccca71117bc P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:34.424333 15923 ts_tablet_manager.cc:1888] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.424396 15923 log.cc:1196] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ea1af845a8e4f39a973d1f8ef691762
I20240501 14:09:34.424974 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 5e22abf2e2e544eb9ac8dccca71117bc (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.425099 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.425173 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e22abf2e2e544eb9ac8dccca71117bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.425263 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet c7fc40ff90834e84a781cd08a45a33f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.425333 15925 tablet_replica.cc:330] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.425462 15925 raft_consensus.cc:2229] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.425541 15422 raft_consensus.cc:482] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.425621 15422 raft_consensus.cc:504] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.425722 15925 raft_consensus.cc:2258] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.425855 16150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5489eab5b04751bc8b3e5fb9e8cf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.425971 15422 leader_election.cc:288] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:34.426116 16131 raft_consensus.cc:348] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.426223 16131 raft_consensus.cc:374] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.426254 16131 raft_consensus.cc:729] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.426290 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet c7fc40ff90834e84a781cd08a45a33f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:34.426363 16131 consensus_queue.cc:260] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.426429 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0653b46a484e74b3ee0b670ade07cf" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:34.426606 16131 ts_tablet_manager.cc:1398] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:34.426764 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0653b46a484e74b3ee0b670ade07cf" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:34.426898 15221 raft_consensus.cc:2454] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:34.427160 15863 ts_tablet_manager.cc:1398] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:34.427212 15023 leader_election.cc:302] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:34.427410 15425 raft_consensus.cc:2781] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.427454 15425 raft_consensus.cc:482] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.427486 15425 raft_consensus.cc:3037] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.427513 15925 ts_tablet_manager.cc:1875] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.428247 16150 tablet_copy_client.cc:783] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.428390 16150 tablet_copy_client.cc:647] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.428402 15425 raft_consensus.cc:504] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.428740 15425 leader_election.cc:288] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:34.429364 16150 tablet_copy_client.cc:523] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.429643 15925 ts_tablet_manager.cc:1888] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.429705 15925 log.cc:1196] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ea1af845a8e4f39a973d1f8ef691762
I20240501 14:09:34.429947 15925 ts_tablet_manager.cc:1909] T 1ea1af845a8e4f39a973d1f8ef691762 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:34.430510 15923 ts_tablet_manager.cc:1909] T 1ea1af845a8e4f39a973d1f8ef691762 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:34.431185 15923 tablet_replica.cc:330] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:34.431318 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:34.431350 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 1ea1af845a8e4f39a973d1f8ef691762 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.431367 15923 raft_consensus.cc:2229] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.431473 16106 tablet_replica.cc:330] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:34.431565 16106 raft_consensus.cc:2229] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:34.431658 16106 raft_consensus.cc:2258] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:34.431695 15925 tablet_replica.cc:330] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.431670 15923 raft_consensus.cc:2258] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.431824 15925 raft_consensus.cc:2229] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.431931 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 1ea1af845a8e4f39a973d1f8ef691762 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.431999 15925 raft_consensus.cc:2258] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.432432 15925 ts_tablet_manager.cc:1875] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:09:34.432579 15024 leader_election.cc:341] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.432885 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0653b46a484e74b3ee0b670ade07cf" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:34.432977 15923 ts_tablet_manager.cc:1875] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.433470 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0653b46a484e74b3ee0b670ade07cf" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:34.433583 15221 raft_consensus.cc:3037] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.434155 15925 ts_tablet_manager.cc:1888] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.434211 15925 log.cc:1196] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/5e22abf2e2e544eb9ac8dccca71117bc
I20240501 14:09:34.434495 15925 ts_tablet_manager.cc:1909] T 5e22abf2e2e544eb9ac8dccca71117bc P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:34.434584 15221 raft_consensus.cc:2454] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:34.434931 15023 leader_election.cc:302] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:34.435119 15925 tablet_replica.cc:330] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.435134 15422 raft_consensus.cc:2781] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.435168 15923 ts_tablet_manager.cc:1888] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.435207 15422 raft_consensus.cc:686] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:34.435228 15923 log.cc:1196] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7fc40ff90834e84a781cd08a45a33f8
I20240501 14:09:34.435202 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 5e22abf2e2e544eb9ac8dccca71117bc (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.435245 15925 raft_consensus.cc:2229] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.435349 15925 raft_consensus.cc:2258] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.435364 15422 consensus_queue.cc:237] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.435477 15923 ts_tablet_manager.cc:1909] T c7fc40ff90834e84a781cd08a45a33f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:34.435817 15925 ts_tablet_manager.cc:1875] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.436103 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet c7fc40ff90834e84a781cd08a45a33f8 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.436631 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea1af845a8e4f39a973d1f8ef691762 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:34.436657 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet c7fc40ff90834e84a781cd08a45a33f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:34.436818 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e22abf2e2e544eb9ac8dccca71117bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:34.436987 15922 tablet_replica.cc:330] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:34.437134 15922 raft_consensus.cc:2229] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.437289 15849 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.437266 15922 raft_consensus.cc:2258] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.437542 15849 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:34.437683 15849 ts_tablet_manager.cc:1367] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.095s user 0.025s sys 0.000s
I20240501 14:09:34.437891 15925 ts_tablet_manager.cc:1888] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.438076 15922 ts_tablet_manager.cc:1875] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.438177 15849 raft_consensus.cc:348] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:34.438288 15849 raft_consensus.cc:729] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
W20240501 14:09:34.438340 15024 leader_election.cc:341] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.438422 16150 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:34.438396 15849 consensus_queue.cc:260] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:34.438834 15849 ts_tablet_manager.cc:1398] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:34.439446 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-d64234effb3446bba1ff82bdf37acaa7 received from {username='slave'} at 127.6.82.8:58949
I20240501 14:09:34.439556 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session f23f426391724d6688aafffe7ee017d8-d64234effb3446bba1ff82bdf37acaa7 on tablet d64234effb3446bba1ff82bdf37acaa7 with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:34.440213 15922 ts_tablet_manager.cc:1888] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.440268 15922 log.cc:1196] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ea1af845a8e4f39a973d1f8ef691762
I20240501 14:09:34.440503 15922 ts_tablet_manager.cc:1909] T 1ea1af845a8e4f39a973d1f8ef691762 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:34.441310 15221 raft_consensus.cc:1203] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.441511 15925 log.cc:1196] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7fc40ff90834e84a781cd08a45a33f8
I20240501 14:09:34.441574 12442 catalog_manager.cc:5484] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:34.441771 15925 ts_tablet_manager.cc:1909] T c7fc40ff90834e84a781cd08a45a33f8 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:34.442338 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet c7fc40ff90834e84a781cd08a45a33f8 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.444469 15922 tablet_replica.cc:330] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:34.444594 15922 raft_consensus.cc:2229] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.444758 15922 raft_consensus.cc:2258] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.445205 15922 ts_tablet_manager.cc:1875] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.445647 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 1ea1af845a8e4f39a973d1f8ef691762 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.438635 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-1ea1af845a8e4f39a973d1f8ef691762 received from {username='slave'} at 127.6.82.6:42689
I20240501 14:09:34.445788 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session ab90e77f0027449bb3155980ca95593c-1ea1af845a8e4f39a973d1f8ef691762 on tablet 1ea1af845a8e4f39a973d1f8ef691762 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:34.447505 15922 ts_tablet_manager.cc:1888] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.447609 15922 log.cc:1196] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/5e22abf2e2e544eb9ac8dccca71117bc
I20240501 14:09:34.447944 15922 ts_tablet_manager.cc:1909] T 5e22abf2e2e544eb9ac8dccca71117bc P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:34.447954 15425 raft_consensus.cc:482] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.448082 15425 raft_consensus.cc:504] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.448426 15425 leader_election.cc:288] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:34.448475 15922 tablet_replica.cc:330] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:34.448581 15922 raft_consensus.cc:2229] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.448634 15684 raft_consensus.cc:482] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.448720 15922 raft_consensus.cc:2258] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.448726 15684 raft_consensus.cc:504] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.449087 15684 leader_election.cc:288] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:34.449162 15922 ts_tablet_manager.cc:1875] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.449245 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41acbf912c4dc0a68fe5b99df2f84b" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:34.449919 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188ce319682147ddaad30aae7232a311" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:34.449916 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188ce319682147ddaad30aae7232a311" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:34.450026 15089 raft_consensus.cc:2454] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
W20240501 14:09:34.450330 15025 leader_election.cc:341] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:09:34.450466 12914 leader_election.cc:341] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.450666 12915 leader_election.cc:302] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c; no voters: 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:34.450978 15922 ts_tablet_manager.cc:1888] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.451102 15922 log.cc:1196] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/c7fc40ff90834e84a781cd08a45a33f8
I20240501 14:09:34.451134 15684 raft_consensus.cc:2781] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.451161 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 5e22abf2e2e544eb9ac8dccca71117bc (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.451203 15684 raft_consensus.cc:482] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.451272 15684 raft_consensus.cc:3037] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.451581 15922 ts_tablet_manager.cc:1909] T c7fc40ff90834e84a781cd08a45a33f8 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:34.451656 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41acbf912c4dc0a68fe5b99df2f84b" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
W20240501 14:09:34.452083 15026 leader_election.cc:341] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.452173 15026 leader_election.cc:302] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab90e77f0027449bb3155980ca95593c; no voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8
I20240501 14:09:34.452258 15684 raft_consensus.cc:504] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.452515 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet c7fc40ff90834e84a781cd08a45a33f8 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.452646 15684 leader_election.cc:288] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:34.452982 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188ce319682147ddaad30aae7232a311" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:34.453022 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188ce319682147ddaad30aae7232a311" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:34.453095 15089 raft_consensus.cc:3037] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:34.453401 12914 leader_election.cc:341] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.453570 15425 raft_consensus.cc:2726] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:34.454039 15089 raft_consensus.cc:2454] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:34.454347 12915 leader_election.cc:302] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c; no voters: 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:34.454504 15683 raft_consensus.cc:2781] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.454595 15683 raft_consensus.cc:686] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:34.454766 15683 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.454911 15833 tablet_copy_client.cc:783] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.455060 15833 tablet_copy_client.cc:647] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.461910 15833 tablet_copy_client.cc:523] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.463069 16134 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:34.463438 15833 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:34.463510 16106 ts_tablet_manager.cc:1875] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.464138 16134 tablet_bootstrap.cc:654] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.465731 16134 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:34.465725 16106 ts_tablet_manager.cc:1888] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.465804 16106 log.cc:1196] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/1ea1af845a8e4f39a973d1f8ef691762
I20240501 14:09:34.466379 16134 ts_tablet_manager.cc:1367] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:34.469446 16134 raft_consensus.cc:348] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.469645 16134 raft_consensus.cc:374] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.469729 16134 raft_consensus.cc:729] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.470011 14958 raft_consensus.cc:1203] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.470324 16134 consensus_queue.cc:260] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.471746 16106 ts_tablet_manager.cc:1909] T 1ea1af845a8e4f39a973d1f8ef691762 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:34.472577 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 1ea1af845a8e4f39a973d1f8ef691762 (table rebalance_test_table_6 [id=5840ee436a6f4345a0c51eceb1e993e0]) successfully deleted
I20240501 14:09:34.473611 16134 ts_tablet_manager.cc:1398] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:34.474260 12585 consensus_queue.cc:237] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:34.475665 12978 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.475834 12718 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:34.476195 15655 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:34.476753 12520 consensus_peers.cc:485] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a99b39e354144e48eee75f9f49a09fc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.477015 14722 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:34.477387 12438 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:34.477699 16138 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:34.477785 16138 ts_tablet_manager.cc:1367] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.000s
I20240501 14:09:34.478209 16138 raft_consensus.cc:348] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.478291 16138 raft_consensus.cc:374] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.478322 16138 raft_consensus.cc:729] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.478425 16138 consensus_queue.cc:260] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.478741 16138 ts_tablet_manager.cc:1398] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.479743 16165 ts_tablet_manager.cc:902] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: Initiating tablet copy from peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:34.480402 16165 tablet_copy_client.cc:308] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.5:33177
I20240501 14:09:34.481086 14978 tablet_copy_service.cc:140] P 8a13c028ba8a4a659b74af6c265df95e: Received BeginTabletCopySession request for tablet 5dd29fa5dc2b4408878ceb3c12a57098 from peer a6e94562bee149c2ace25672f0f9bf50 ({username='slave'} at 127.6.82.4:34811)
I20240501 14:09:34.481220 14978 tablet_copy_service.cc:161] P 8a13c028ba8a4a659b74af6c265df95e: Beginning new tablet copy session on tablet 5dd29fa5dc2b4408878ceb3c12a57098 from peer a6e94562bee149c2ace25672f0f9bf50 at {username='slave'} at 127.6.82.4:34811: session id = a6e94562bee149c2ace25672f0f9bf50-5dd29fa5dc2b4408878ceb3c12a57098
I20240501 14:09:34.482897 15476 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:34.487005 15425 raft_consensus.cc:482] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.487103 15425 raft_consensus.cc:504] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.487476 15425 leader_election.cc:288] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:34.487918 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:34.488106 12718 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
W20240501 14:09:34.488260 15024 leader_election.cc:341] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.488745 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:34.488852 12585 raft_consensus.cc:2454] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:34.489871 12436 catalog_manager.cc:5484] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:34.490723 15025 leader_election.cc:302] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c; no voters: 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:34.491045 15425 raft_consensus.cc:2781] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.491120 15425 raft_consensus.cc:482] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.491160 15425 raft_consensus.cc:3037] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.491897 14958 raft_consensus.cc:1203] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.492170 15425 raft_consensus.cc:504] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.492511 15425 leader_election.cc:288] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:34.493116 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:34.493223 12585 raft_consensus.cc:3037] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.493731 12978 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:34.494290 12585 raft_consensus.cc:2454] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:34.494463 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:34.494688 15025 leader_election.cc:302] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c; no voters:
W20240501 14:09:34.494813 15024 leader_election.cc:341] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:34.495828 15833 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.496091 15833 tablet_bootstrap.cc:492] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:34.496148 14978 tablet_copy_source_session.cc:205] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.496258 15833 ts_tablet_manager.cc:1367] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240501 14:09:34.497035 16132 tablet_bootstrap.cc:492] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:34.497112 16132 ts_tablet_manager.cc:1367] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.137s user 0.002s sys 0.000s
I20240501 14:09:34.497431 15833 raft_consensus.cc:348] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:34.497545 15833 raft_consensus.cc:729] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:34.497527 16132 raft_consensus.cc:348] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.497675 16132 raft_consensus.cc:374] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.497710 16132 raft_consensus.cc:729] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.497716 15833 consensus_queue.cc:260] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:34.497843 16132 consensus_queue.cc:260] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:34.498040 15833 ts_tablet_manager.cc:1398] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.498184 16132 ts_tablet_manager.cc:1398] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.498361 16132 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:34.499347 16132 tablet_bootstrap.cc:654] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.501191 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-25b5c40793b348bbb0e5251c8d553e30 received from {username='slave'} at 127.6.82.5:59069
I20240501 14:09:34.501201 16165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dd29fa5dc2b4408878ceb3c12a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.501289 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-25b5c40793b348bbb0e5251c8d553e30 on tablet 25b5c40793b348bbb0e5251c8d553e30 with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:34.503298 16150 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.503326 16165 tablet_copy_client.cc:783] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.503491 16165 tablet_copy_client.cc:647] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.503824 16150 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:34.504495 16150 ts_tablet_manager.cc:1367] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.066s user 0.023s sys 0.006s
I20240501 14:09:34.504777 15425 raft_consensus.cc:2781] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.504877 15425 raft_consensus.cc:686] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:34.505018 15425 consensus_queue.cc:237] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.505370 16150 raft_consensus.cc:348] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:34.505534 16150 raft_consensus.cc:729] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:34.505734 15655 raft_consensus.cc:482] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.505812 15655 raft_consensus.cc:504] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.506137 15655 leader_election.cc:288] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:34.506414 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41acbf912c4dc0a68fe5b99df2f84b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:34.506512 15352 raft_consensus.cc:2454] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:34.506493 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41acbf912c4dc0a68fe5b99df2f84b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:34.506603 15089 raft_consensus.cc:2454] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:34.506842 16150 consensus_queue.cc:260] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:34.507153 12523 leader_election.cc:302] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:34.507520 16165 tablet_copy_client.cc:523] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.507846 15655 raft_consensus.cc:2781] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:34.507898 15655 raft_consensus.cc:482] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:34.507930 15655 raft_consensus.cc:3037] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.508689 16150 ts_tablet_manager.cc:1398] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:34.508816 15655 raft_consensus.cc:504] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.509126 16165 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:34.509140 15655 leader_election.cc:288] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:34.509508 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41acbf912c4dc0a68fe5b99df2f84b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:34.509610 15089 raft_consensus.cc:3037] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.509816 12438 catalog_manager.cc:5484] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:34.510177 16132 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:34.510278 16132 ts_tablet_manager.cc:1367] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:34.510497 15089 raft_consensus.cc:2454] T 0c41acbf912c4dc0a68fe5b99df2f84b P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:34.510836 16132 raft_consensus.cc:348] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.510943 16132 raft_consensus.cc:374] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.510975 16132 raft_consensus.cc:729] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.511085 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-bd5489eab5b04751bc8b3e5fb9e8cf73 received from {username='slave'} at 127.6.82.8:58949
I20240501 14:09:34.511086 16132 consensus_queue.cc:260] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:34.511155 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session f23f426391724d6688aafffe7ee017d8-bd5489eab5b04751bc8b3e5fb9e8cf73 on tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:34.511373 16132 ts_tablet_manager.cc:1398] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:34.511528 16132 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:34.512310 12522 leader_election.cc:302] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:34.512475 15655 raft_consensus.cc:2781] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:34.512553 16132 tablet_bootstrap.cc:654] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:34.512550 15655 raft_consensus.cc:686] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:34.512676 15655 consensus_queue.cc:237] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:34.513908 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c41acbf912c4dc0a68fe5b99df2f84b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:34.514039 15352 raft_consensus.cc:3037] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.514647 14958 raft_consensus.cc:1203] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.515131 15352 raft_consensus.cc:2454] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:34.515409 16132 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:34.515496 16132 ts_tablet_manager.cc:1367] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:34.516173 16132 raft_consensus.cc:348] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.516273 16132 raft_consensus.cc:374] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:34.516305 16132 raft_consensus.cc:729] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:34.516773 16132 consensus_queue.cc:260] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.517005 12438 catalog_manager.cc:5484] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:34.517145 16132 ts_tablet_manager.cc:1398] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:34.518536 15221 consensus_queue.cc:237] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:34.521111 15089 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:34.521423 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41a1a3342cae4af480f1e65164493219. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:34.521535 15352 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:34.521684 15572 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:34.522003 15572 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:34.523705 15089 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:34.524087 15352 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:34.523386 16184 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:34.526249 12436 catalog_manager.cc:5484] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:34.531795 15421 raft_consensus.cc:482] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:34.531908 15421 raft_consensus.cc:504] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:34.532323 15421 leader_election.cc:288] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:34.532739 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:34.532905 12585 raft_consensus.cc:2379] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a13c028ba8a4a659b74af6c265df95e in current term 1: Already voted for candidate ab90e77f0027449bb3155980ca95593c in this term.
I20240501 14:09:34.533655 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:34.534188 14893 leader_election.cc:302] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e; no voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c
I20240501 14:09:34.534442 15421 raft_consensus.cc:3037] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.535629 15421 raft_consensus.cc:2726] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:34.536238 15352 raft_consensus.cc:1203] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.542477 16165 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.542901 16165 tablet_bootstrap.cc:492] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap complete.
I20240501 14:09:34.543103 16165 ts_tablet_manager.cc:1367] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.002s
I20240501 14:09:34.543587 16165 raft_consensus.cc:348] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:34.543741 16165 raft_consensus.cc:729] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: LEARNER
I20240501 14:09:34.543864 16165 consensus_queue.cc:260] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:34.544242 16165 ts_tablet_manager.cc:1398] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:34.545814 14978 tablet_copy_service.cc:342] P 8a13c028ba8a4a659b74af6c265df95e: Request end of tablet copy session a6e94562bee149c2ace25672f0f9bf50-5dd29fa5dc2b4408878ceb3c12a57098 received from {username='slave'} at 127.6.82.4:34811
I20240501 14:09:34.545954 14978 tablet_copy_service.cc:433] P 8a13c028ba8a4a659b74af6c265df95e: ending tablet copy session a6e94562bee149c2ace25672f0f9bf50-5dd29fa5dc2b4408878ceb3c12a57098 on tablet 5dd29fa5dc2b4408878ceb3c12a57098 with peer a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:34.546272 15863 ts_tablet_manager.cc:902] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:34.546769 15863 tablet_copy_client.cc:308] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:34.547178 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet d2631324f66d454697b895ecdd227276 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:42689)
I20240501 14:09:34.547263 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet d2631324f66d454697b895ecdd227276 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:42689: session id = ab90e77f0027449bb3155980ca95593c-d2631324f66d454697b895ecdd227276
I20240501 14:09:34.548226 12999 tablet_copy_source_session.cc:205] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.549124 15863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2631324f66d454697b895ecdd227276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.551201 15863 tablet_copy_client.cc:783] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.551342 15863 tablet_copy_client.cc:647] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.553262 15863 tablet_copy_client.cc:523] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.553994 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:34.555486 15863 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:34.558698 15221 raft_consensus.cc:1261] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:34.559201 14958 raft_consensus.cc:3037] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.560416 14958 raft_consensus.cc:1261] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:34.560832 15422 consensus_queue.cc:1035] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.560998 15425 consensus_queue.cc:1035] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.563597 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 2a0653b46a484e74b3ee0b670ade07cf (table rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc]). Expected version 1 got 0
I20240501 14:09:34.563802 15470 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:34.564107 15470 tablet.cc:1620] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:34.564137 12442 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 2a0653b46a484e74b3ee0b670ade07cf (table rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc]). Expected version 1 got 0
I20240501 14:09:34.566089 15647 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:34.566370 15647 tablet.cc:1620] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:34.568971 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc] alter complete (version 1)
I20240501 14:09:34.570519 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 2a0653b46a484e74b3ee0b670ade07cf (table rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc]). Expected version 1 got 0
I20240501 14:09:34.571200 16212 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:34.571542 16212 tablet.cc:1620] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:34.601604 15863 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:34.601951 15863 tablet_bootstrap.cc:492] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:34.602152 15863 ts_tablet_manager.cc:1367] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.000s
I20240501 14:09:34.602591 15863 raft_consensus.cc:348] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:34.602710 15863 raft_consensus.cc:729] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:34.602839 15863 consensus_queue.cc:260] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:34.603168 15863 ts_tablet_manager.cc:1398] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:34.603808 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-d2631324f66d454697b895ecdd227276 received from {username='slave'} at 127.6.82.6:42689
I20240501 14:09:34.603906 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session ab90e77f0027449bb3155980ca95593c-d2631324f66d454697b895ecdd227276 on tablet d2631324f66d454697b895ecdd227276 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:34.661826 15476 raft_consensus.cc:1050] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:34.662482 15476 consensus_queue.cc:237] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:34.664060 15352 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.664142 12849 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.664152 12980 raft_consensus.cc:1261] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.664525 14722 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.664736 15655 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.664917 16290 consensus_queue.cc:1035] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.667379 15655 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:34.667943 15352 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:34.668004 12849 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:34.669593 12980 raft_consensus.cc:2932] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:34.671172 12442 catalog_manager.cc:5484] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:34.673084 12437 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:34.673903 14452 raft_consensus.cc:1050] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:34.674434 14452 consensus_queue.cc:237] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:34.676079 12586 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.676208 15089 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.676741 15980 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.676932 16107 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.678439 12980 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.678915 16107 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.681159 15980 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.683203 15089 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.683372 15470 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.683678 15470 tablet.cc:1620] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:34.683972 16212 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.684289 16212 tablet.cc:1620] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:34.687055 12437 catalog_manager.cc:5484] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.688587 15647 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:34.688896 15647 tablet.cc:1620] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:34.689755 12586 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.693863 12718 consensus_queue.cc:237] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:34.694145 12436 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=cf4a10270c4642cbb194a987fc3124cc] alter complete (version 2)
I20240501 14:09:34.696484 15089 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.696652 12586 raft_consensus.cc:1261] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.697304 14452 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.697435 16304 consensus_queue.cc:1035] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:09:34.697738 16107 consensus_queue.cc:1159] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a6e94562bee149c2ace25672f0f9bf50" responder_term: 1 status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240501 14:09:34.699242 16107 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.699658 12585 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.701864 15089 raft_consensus.cc:2932] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.702419 12444 catalog_manager.cc:5484] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:34.702515 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6305f3f7cb8846bf843d2d753fd1ae07 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.705559 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.705780 15925 tablet_replica.cc:330] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.705912 15925 raft_consensus.cc:2229] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.706041 15925 pending_rounds.cc:62] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Trying to abort 1 pending ops.
I20240501 14:09:34.706101 15925 pending_rounds.cc:68] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022889265865216000 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6305f3f7cb8846bf843d2d753fd1ae07" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } } }
I20240501 14:09:34.706238 15925 raft_consensus.cc:2894] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:09:34.706295 15925 raft_consensus.cc:2866] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:34.706331 15925 raft_consensus.cc:2258] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.707018 15925 ts_tablet_manager.cc:1875] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.708967 15925 ts_tablet_manager.cc:1888] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:34.709035 15925 log.cc:1196] T 6305f3f7cb8846bf843d2d753fd1ae07 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/6305f3f7cb8846bf843d2d753fd1ae07
I20240501 14:09:34.709498 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 6305f3f7cb8846bf843d2d753fd1ae07 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:34.719489 15089 raft_consensus.cc:1203] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.775120 15200 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.775763 12567 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:34.780414 12958 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 14:09:34.785951 12698 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:34.786249 15332 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:09:34.788079 14938 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:34.803913 12437 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:34.807464 16290 raft_consensus.cc:1050] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:34.807934 16290 consensus_queue.cc:237] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:34.809455 14958 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.809593 12849 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.810061 12718 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.810034 14722 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.810659 15476 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.810945 12829 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:34.811003 15476 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.816047 15470 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.819139 16212 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.823123 15647 tablet.cc:1722] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:34.822600 12437 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=cf4a10270c4642cbb194a987fc3124cc] alter complete (version 3)
I20240501 14:09:34.831899 14722 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.833261 15069 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:34.834514 14958 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.834951 12848 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.835063 12437 catalog_manager.cc:5484] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:34.845093 16290 raft_consensus.cc:1050] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:34.845542 16290 consensus_queue.cc:237] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:34.845810 14958 raft_consensus.cc:3037] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:34.847096 12716 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.847308 15683 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.847397 12980 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.847903 15655 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.848467 15655 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.849109 15089 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.850617 16290 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.853499 16290 raft_consensus.cc:1050] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:34.854204 16290 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:34.856251 12848 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.857025 16290 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.857450 15476 raft_consensus.cc:1050] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:34.857825 15476 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:34.859525 15221 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.859757 12716 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.860321 15476 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.860471 14722 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.861122 15352 raft_consensus.cc:1203] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.862075 12978 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.862255 14957 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.862648 15655 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.863029 15655 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.864634 12849 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.865191 16290 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.867589 15655 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.869539 12716 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.869601 15089 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.872155 12441 catalog_manager.cc:5484] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:34.873443 16290 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.875370 14722 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.875597 12849 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.875967 12585 consensus_queue.cc:237] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:34.876715 12980 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.877350 12849 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.878278 12978 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.880602 12440 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.882200 15089 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.882572 15655 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.884891 15352 raft_consensus.cc:1203] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.885151 15476 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.886766 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afa5ebe09c99435fb69ceb527ab4734a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.887548 15221 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.887709 12717 raft_consensus.cc:1261] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.888554 12717 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.888808 12441 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.889235 14722 consensus_queue.cc:1035] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.892102 12716 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.895260 12436 catalog_manager.cc:5484] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:34.897783 15089 raft_consensus.cc:2932] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:34.900586 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5ebe09c99435fb69ceb527ab4734a with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.900821 15925 tablet_replica.cc:330] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.900964 15925 raft_consensus.cc:2229] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.901121 15925 raft_consensus.cc:2258] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.901201 12585 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:34.901890 15925 ts_tablet_manager.cc:1875] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.904047 15221 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.904047 12849 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.904424 16367 ts_tablet_manager.cc:902] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:34.904424 15925 ts_tablet_manager.cc:1888] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:34.904495 15925 log.cc:1196] T afa5ebe09c99435fb69ceb527ab4734a P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/afa5ebe09c99435fb69ceb527ab4734a
I20240501 14:09:34.904631 14958 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.905089 16367 tablet_copy_client.cc:308] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:34.905565 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 1a99b39e354144e48eee75f9f49a09fc from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:51545)
I20240501 14:09:34.905666 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 1a99b39e354144e48eee75f9f49a09fc from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:51545: session id = 66e2dc34330848c3b8c2dd15898d34c8-1a99b39e354144e48eee75f9f49a09fc
I20240501 14:09:34.906435 15655 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.906502 15684 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.906694 12607 tablet_copy_source_session.cc:205] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:34.907279 14957 raft_consensus.cc:1203] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.907532 14722 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.907667 16367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99b39e354144e48eee75f9f49a09fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:34.909794 15684 raft_consensus.cc:1050] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:34.909767 14722 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.910024 16367 tablet_copy_client.cc:783] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:34.910171 16367 tablet_copy_client.cc:647] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:34.910234 15684 consensus_queue.cc:237] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:34.911124 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet afa5ebe09c99435fb69ceb527ab4734a (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:34.912163 12849 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.913599 12718 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.913733 15222 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.914408 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7693e3e204634676a758726bb6d3d2ce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.916172 12586 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:34.916110 12441 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.917809 12848 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.918046 12586 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.918434 14674 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.918831 14674 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.920802 12585 consensus_queue.cc:237] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:34.920938 15222 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.921346 14674 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.922212 12978 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.922276 14958 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.922617 14957 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.922957 15655 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.923064 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7693e3e204634676a758726bb6d3d2ce with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:34.923465 15923 tablet_replica.cc:330] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:34.923604 15923 raft_consensus.cc:2229] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.923741 15923 raft_consensus.cc:2258] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.923774 12436 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:09:34.923907 12436 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 14:09:34.924607 15923 ts_tablet_manager.cc:1875] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.924587 14722 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.924880 14722 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.926774 12718 raft_consensus.cc:1261] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.927032 12440 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.927134 12440 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:34.927798 14722 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.928529 15923 ts_tablet_manager.cc:1888] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:34.928606 15923 log.cc:1196] T 7693e3e204634676a758726bb6d3d2ce P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/7693e3e204634676a758726bb6d3d2ce
I20240501 14:09:34.929105 14722 consensus_queue.cc:1035] T 66436017fd044f0b919629933ea56e9c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.929201 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66436017fd044f0b919629933ea56e9c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.929376 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 7693e3e204634676a758726bb6d3d2ce (table rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]) successfully deleted
I20240501 14:09:34.929670 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet afe9066fbac74c29b51fce22a672b65b
I20240501 14:09:34.930032 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 59c4576fd61a4209ab979f8d35243e94
I20240501 14:09:34.930244 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet b547fcb34f12402abccdb23678c95e26
I20240501 14:09:34.930245 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet afe9066fbac74c29b51fce22a672b65b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:34.930171 12978 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.930661 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet b547fcb34f12402abccdb23678c95e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:34.931007 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet afe9066fbac74c29b51fce22a672b65b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:34.931075 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet b547fcb34f12402abccdb23678c95e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:34.931308 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c4576fd61a4209ab979f8d35243e94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:34.929634 14722 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.931818 12718 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.932042 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet b547fcb34f12402abccdb23678c95e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.932173 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c4576fd61a4209ab979f8d35243e94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:34.932190 15925 tablet_replica.cc:330] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.932340 15925 raft_consensus.cc:2229] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.932412 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet afe9066fbac74c29b51fce22a672b65b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:34.932512 15925 raft_consensus.cc:2258] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.932540 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c4576fd61a4209ab979f8d35243e94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:34.932626 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet b547fcb34f12402abccdb23678c95e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:34.932754 15922 tablet_replica.cc:330] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:34.932871 15922 raft_consensus.cc:2229] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.933032 15922 raft_consensus.cc:2258] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.933274 15923 tablet_replica.cc:330] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:34.933393 15923 raft_consensus.cc:2229] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.933512 15923 raft_consensus.cc:2258] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.933529 15922 ts_tablet_manager.cc:1875] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.934044 15923 ts_tablet_manager.cc:1875] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.934474 15925 ts_tablet_manager.cc:1875] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.935652 15922 ts_tablet_manager.cc:1888] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.935735 15922 log.cc:1196] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/afe9066fbac74c29b51fce22a672b65b
I20240501 14:09:34.936033 15922 ts_tablet_manager.cc:1909] T afe9066fbac74c29b51fce22a672b65b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:34.936201 15923 ts_tablet_manager.cc:1888] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.936275 15923 log.cc:1196] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/afe9066fbac74c29b51fce22a672b65b
I20240501 14:09:34.936486 15923 ts_tablet_manager.cc:1909] T afe9066fbac74c29b51fce22a672b65b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:34.936595 15922 tablet_replica.cc:330] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:34.936656 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet afe9066fbac74c29b51fce22a672b65b (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.936713 15922 raft_consensus.cc:2229] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.936889 15922 raft_consensus.cc:2258] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.936928 15925 ts_tablet_manager.cc:1888] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:34.936985 15925 log.cc:1196] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/b547fcb34f12402abccdb23678c95e26
I20240501 14:09:34.937176 15923 tablet_replica.cc:330] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:34.937227 15925 ts_tablet_manager.cc:1909] T b547fcb34f12402abccdb23678c95e26 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:34.937301 15923 raft_consensus.cc:2229] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.937386 15923 raft_consensus.cc:2258] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.937428 15922 ts_tablet_manager.cc:1875] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.937716 15923 ts_tablet_manager.cc:1875] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.937752 15925 tablet_replica.cc:330] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:34.937870 15925 raft_consensus.cc:2229] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.937876 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet b547fcb34f12402abccdb23678c95e26 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.937978 15925 raft_consensus.cc:2258] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.938410 15925 ts_tablet_manager.cc:1875] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.938877 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet afe9066fbac74c29b51fce22a672b65b (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.939267 15922 ts_tablet_manager.cc:1888] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.939133 14958 raft_consensus.cc:2932] T 66436017fd044f0b919629933ea56e9c P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.939340 15922 log.cc:1196] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/59c4576fd61a4209ab979f8d35243e94
I20240501 14:09:34.939540 15923 ts_tablet_manager.cc:1888] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.939581 15922 ts_tablet_manager.cc:1909] T 59c4576fd61a4209ab979f8d35243e94 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:34.939599 15923 log.cc:1196] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/59c4576fd61a4209ab979f8d35243e94
I20240501 14:09:34.939841 15923 ts_tablet_manager.cc:1909] T 59c4576fd61a4209ab979f8d35243e94 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:34.940138 15922 tablet_replica.cc:330] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:34.940253 15922 raft_consensus.cc:2229] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.940251 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 59c4576fd61a4209ab979f8d35243e94 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.940814 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 59c4576fd61a4209ab979f8d35243e94 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.941066 16379 tablet_replica.cc:330] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:34.941695 12440 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 2a0653b46a484e74b3ee0b670ade07cf
I20240501 14:09:34.941896 12445 catalog_manager.cc:5484] T 66436017fd044f0b919629933ea56e9c P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.942445 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0653b46a484e74b3ee0b670ade07cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:34.942641 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0653b46a484e74b3ee0b670ade07cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:34.942865 16367 tablet_copy_client.cc:523] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:34.944576 16367 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:34.944922 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 66436017fd044f0b919629933ea56e9c with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:34.944880 12436 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.945273 16379 raft_consensus.cc:2229] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.945410 16379 raft_consensus.cc:2258] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.945477 14957 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:34.945870 16379 ts_tablet_manager.cc:1875] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.949054 16375 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.950340 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet d178c3291efc44369e3adb921ccc6517 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:34.951114 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d178c3291efc44369e3adb921ccc6517 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.951500 15221 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.951671 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0653b46a484e74b3ee0b670ade07cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:34 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:34.951844 16106 tablet_replica.cc:330] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:34.951951 16106 raft_consensus.cc:2229] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:34.952137 16106 raft_consensus.cc:2258] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.952631 15925 ts_tablet_manager.cc:1888] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.952700 15925 log.cc:1196] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/59c4576fd61a4209ab979f8d35243e94
I20240501 14:09:34.952822 16106 ts_tablet_manager.cc:1875] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.952983 15925 ts_tablet_manager.cc:1909] T 59c4576fd61a4209ab979f8d35243e94 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:34.954222 12848 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.955008 16380 tablet_replica.cc:330] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:34.955163 16380 raft_consensus.cc:2229] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.955314 16380 raft_consensus.cc:2258] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.956061 12440 catalog_manager.cc:5484] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.956848 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 59c4576fd61a4209ab979f8d35243e94 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.957957 16379 ts_tablet_manager.cc:1888] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:34.958027 16379 log.cc:1196] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/afe9066fbac74c29b51fce22a672b65b
I20240501 14:09:34.958351 16379 ts_tablet_manager.cc:1909] T afe9066fbac74c29b51fce22a672b65b P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:34.959410 15425 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.959859 15655 consensus_queue.cc:1035] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.960054 16379 tablet_replica.cc:330] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:34.960302 16379 raft_consensus.cc:2229] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.960460 16379 raft_consensus.cc:2258] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.961133 16379 ts_tablet_manager.cc:1875] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.961349 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet afe9066fbac74c29b51fce22a672b65b (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.962591 16106 ts_tablet_manager.cc:1888] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:34.962662 16106 log.cc:1196] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/2a0653b46a484e74b3ee0b670ade07cf
I20240501 14:09:34.962996 16106 ts_tablet_manager.cc:1909] T 2a0653b46a484e74b3ee0b670ade07cf P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:34.963681 12586 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.963907 16379 ts_tablet_manager.cc:1888] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:34.963968 16379 log.cc:1196] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/b547fcb34f12402abccdb23678c95e26
I20240501 14:09:34.964227 16379 ts_tablet_manager.cc:1909] T b547fcb34f12402abccdb23678c95e26 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:34.964761 16379 tablet_replica.cc:330] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:34.964946 16379 raft_consensus.cc:2229] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.964948 15476 consensus_queue.cc:1035] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.965081 16379 raft_consensus.cc:2258] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.965456 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet b547fcb34f12402abccdb23678c95e26 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.966444 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 2a0653b46a484e74b3ee0b670ade07cf (table rebalancer_dynamic_table_new_name [id=cf4a10270c4642cbb194a987fc3124cc]) successfully deleted
I20240501 14:09:34.968506 16380 ts_tablet_manager.cc:1875] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.969270 15922 raft_consensus.cc:2258] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.969383 16379 ts_tablet_manager.cc:1875] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.969861 12978 consensus_queue.cc:237] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:34.971640 15222 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.972357 15683 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.973069 16379 ts_tablet_manager.cc:1888] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:34.973207 16379 log.cc:1196] T 66436017fd044f0b919629933ea56e9c P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/66436017fd044f0b919629933ea56e9c
I20240501 14:09:34.973456 16375 raft_consensus.cc:1050] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:34.973841 16375 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:34.972225 12586 raft_consensus.cc:1261] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:34.974094 15683 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.974675 16375 consensus_queue.cc:1035] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:34.974753 16380 ts_tablet_manager.cc:1888] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:34.974823 16380 log.cc:1196] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/b547fcb34f12402abccdb23678c95e26
I20240501 14:09:34.975684 12586 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.976306 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49f2b0e67d3c43d584ac7583b98caeb9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:34.976711 14958 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.977092 12445 catalog_manager.cc:5484] T 49f2b0e67d3c43d584ac7583b98caeb9 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:34.977381 15222 raft_consensus.cc:2932] T 49f2b0e67d3c43d584ac7583b98caeb9 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:34.977694 12718 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.978106 16375 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.978421 16375 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.979876 12849 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.981238 15684 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.981588 16379 tablet_replica.cc:330] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:34.981616 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f2b0e67d3c43d584ac7583b98caeb9 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:34.981690 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 66436017fd044f0b919629933ea56e9c (table rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]) successfully deleted
I20240501 14:09:34.981997 16379 raft_consensus.cc:2229] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.983101 16379 raft_consensus.cc:2258] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.983800 16379 ts_tablet_manager.cc:1875] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.984246 15089 raft_consensus.cc:1203] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:34.984583 15922 ts_tablet_manager.cc:1875] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:34.987268 16379 ts_tablet_manager.cc:1888] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:34.987423 16379 log.cc:1196] T d178c3291efc44369e3adb921ccc6517 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/d178c3291efc44369e3adb921ccc6517
I20240501 14:09:34.988495 16375 raft_consensus.cc:1050] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:34.988644 15922 ts_tablet_manager.cc:1888] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240501 14:09:34.988713 15922 log.cc:1196] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/b547fcb34f12402abccdb23678c95e26
I20240501 14:09:34.988663 15422 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:34.988976 15922 ts_tablet_manager.cc:1909] T b547fcb34f12402abccdb23678c95e26 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:34.988914 16375 consensus_queue.cc:237] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:34.989370 16382 tablet_replica.cc:330] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:34.989513 16382 raft_consensus.cc:2229] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.989753 16381 tablet_replica.cc:330] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:34.989940 16381 raft_consensus.cc:2229] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.990451 12716 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.991231 12849 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.991580 16389 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.991819 16375 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.990608 15222 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:34.992738 16391 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:34.992846 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet b547fcb34f12402abccdb23678c95e26 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:34.993901 16381 raft_consensus.cc:2258] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.994340 16379 tablet_replica.cc:330] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:34.994499 16379 raft_consensus.cc:2229] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:34.994633 16379 raft_consensus.cc:2258] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:34.995067 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet d178c3291efc44369e3adb921ccc6517 (table rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]) successfully deleted
I20240501 14:09:34.996017 16379 ts_tablet_manager.cc:1875] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:34.999150 16379 ts_tablet_manager.cc:1888] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:34.999441 16379 log.cc:1196] T 49f2b0e67d3c43d584ac7583b98caeb9 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/49f2b0e67d3c43d584ac7583b98caeb9
I20240501 14:09:35.007745 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 49f2b0e67d3c43d584ac7583b98caeb9 (table rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]) successfully deleted
I20240501 14:09:35.009033 16389 raft_consensus.cc:1050] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:35.009626 16389 consensus_queue.cc:237] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:35.010499 16381 ts_tablet_manager.cc:1875] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.011059 15352 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.011812 12585 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.012228 16375 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.012401 16390 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.012714 16381 ts_tablet_manager.cc:1888] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.012843 16382 raft_consensus.cc:2258] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.013516 16382 ts_tablet_manager.cc:1875] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.014330 12717 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.014668 16389 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.015024 16384 ts_tablet_manager.cc:902] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: Initiating tablet copy from peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:35.015906 16382 ts_tablet_manager.cc:1888] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.016053 16382 log.cc:1196] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/2a0653b46a484e74b3ee0b670ade07cf
I20240501 14:09:35.016389 16382 ts_tablet_manager.cc:1909] T 2a0653b46a484e74b3ee0b670ade07cf P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:35.016486 16384 tablet_copy_client.cc:308] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.7:34847
I20240501 14:09:35.017431 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 2a0653b46a484e74b3ee0b670ade07cf (table rebalancer_dynamic_table_new_name [id=cf4a10270c4642cbb194a987fc3124cc]) successfully deleted
I20240501 14:09:35.017256 15242 tablet_copy_service.cc:140] P 66e2dc34330848c3b8c2dd15898d34c8: Received BeginTabletCopySession request for tablet 41a1a3342cae4af480f1e65164493219 from peer 6a3cc0660eed427eaee38164a8c1d4c3 ({username='slave'} at 127.6.82.1:43007)
I20240501 14:09:35.018481 16381 log.cc:1196] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/2a0653b46a484e74b3ee0b670ade07cf
I20240501 14:09:35.018597 15242 tablet_copy_service.cc:161] P 66e2dc34330848c3b8c2dd15898d34c8: Beginning new tablet copy session on tablet 41a1a3342cae4af480f1e65164493219 from peer 6a3cc0660eed427eaee38164a8c1d4c3 at {username='slave'} at 127.6.82.1:43007: session id = 6a3cc0660eed427eaee38164a8c1d4c3-41a1a3342cae4af480f1e65164493219
I20240501 14:09:35.018838 16381 ts_tablet_manager.cc:1909] T 2a0653b46a484e74b3ee0b670ade07cf P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:35.019253 16380 ts_tablet_manager.cc:1909] T b547fcb34f12402abccdb23678c95e26 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:35.019956 15242 tablet_copy_source_session.cc:205] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:35.020673 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet b547fcb34f12402abccdb23678c95e26 (table rebalance_test_table_8 [id=7440582878434b63b8e6e1a506f304d8]) successfully deleted
I20240501 14:09:35.021217 16375 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.022105 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 2a0653b46a484e74b3ee0b670ade07cf (table rebalancer_dynamic_table_new_name [id=cf4a10270c4642cbb194a987fc3124cc]) successfully deleted
I20240501 14:09:35.022064 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:35.023321 15683 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.024981 12585 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.029181 15352 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.030812 16384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a1a3342cae4af480f1e65164493219. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.030961 12440 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.030679 12849 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.032574 12716 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.032665 12717 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.035589 14958 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.041050 12436 catalog_manager.cc:5484] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.041400 12978 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:35.044391 15200 tablet_service.cc:1467] Processing CreateTablet for tablet ba052327b37e4c4592dbf45a080fc5e1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.044683 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba052327b37e4c4592dbf45a080fc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.044763 12696 tablet_service.cc:1467] Processing CreateTablet for tablet bfce6c5fb6344a878935b322b5d8e2d0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.045063 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfce6c5fb6344a878935b322b5d8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.045110 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:35.045814 15332 tablet_service.cc:1467] Processing CreateTablet for tablet ba052327b37e4c4592dbf45a080fc5e1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.046068 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba052327b37e4c4592dbf45a080fc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.047742 14938 tablet_service.cc:1467] Processing CreateTablet for tablet ba052327b37e4c4592dbf45a080fc5e1 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.047979 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba052327b37e4c4592dbf45a080fc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.048197 15333 tablet_service.cc:1467] Processing CreateTablet for tablet bfce6c5fb6344a878935b322b5d8e2d0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.048381 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfce6c5fb6344a878935b322b5d8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.050302 15331 tablet_service.cc:1467] Processing CreateTablet for tablet d521005dcbf74e4fb9a67fe9992ce540 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.050516 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d521005dcbf74e4fb9a67fe9992ce540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.051162 15202 tablet_service.cc:1467] Processing CreateTablet for tablet d521005dcbf74e4fb9a67fe9992ce540 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.051417 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d521005dcbf74e4fb9a67fe9992ce540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.053939 14958 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.055840 15683 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.057315 12849 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.058274 15684 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.059521 16375 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.061304 14958 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.062089 12978 raft_consensus.cc:1203] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:35.062228 14674 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.064399 12718 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.065443 12441 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.065476 12848 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.066035 12849 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.068015 12440 catalog_manager.cc:5484] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.068747 16419 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:35.069862 16419 tablet_bootstrap.cc:654] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.070806 16384 tablet_copy_client.cc:783] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:35.071028 16384 tablet_copy_client.cc:647] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:35.071072 16419 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:35.071147 16419 ts_tablet_manager.cc:1367] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:35.071583 16419 raft_consensus.cc:348] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.071681 16419 raft_consensus.cc:374] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.071710 16419 raft_consensus.cc:729] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.071812 16419 consensus_queue.cc:260] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.072094 16419 ts_tablet_manager.cc:1398] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.072257 16419 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:35.073168 16419 tablet_bootstrap.cc:654] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.073649 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63a76e89d58d48ba9d1b3b98ef6c172d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.074090 16384 tablet_copy_client.cc:523] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:35.074613 15069 tablet_service.cc:1467] Processing CreateTablet for tablet bfce6c5fb6344a878935b322b5d8e2d0 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.074517 15070 tablet_service.cc:1467] Processing CreateTablet for tablet d521005dcbf74e4fb9a67fe9992ce540 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.074695 15980 raft_consensus.cc:1050] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:35.074915 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfce6c5fb6344a878935b322b5d8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.075094 15980 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:35.075510 12978 consensus_queue.cc:237] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:35.077296 12978 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.077663 16384 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:35.077708 16304 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.077857 12718 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.078832 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d521005dcbf74e4fb9a67fe9992ce540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.079195 15683 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.079883 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a76e89d58d48ba9d1b3b98ef6c172d with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.080087 15923 tablet_replica.cc:330] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.080210 15923 raft_consensus.cc:2229] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.080299 12849 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.080358 15923 raft_consensus.cc:2258] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.080667 16304 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.081048 15923 ts_tablet_manager.cc:1875] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.081665 14958 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.082044 16304 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.082962 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]), partition=
I20240501 14:09:35.083132 15923 ts_tablet_manager.cc:1888] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.083199 15923 log.cc:1196] T 63a76e89d58d48ba9d1b3b98ef6c172d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/63a76e89d58d48ba9d1b3b98ef6c172d
I20240501 14:09:35.083204 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089b7a68a30e48f5bbcd6137c245c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.071455 16417 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:35.077705 15352 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.084080 15683 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.084461 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 63a76e89d58d48ba9d1b3b98ef6c172d (table rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]) successfully deleted
I20240501 14:09:35.085007 16417 tablet_bootstrap.cc:654] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.086035 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]), partition=
I20240501 14:09:35.086175 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]), partition=
I20240501 14:09:35.086272 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089b7a68a30e48f5bbcd6137c245c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.086398 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089b7a68a30e48f5bbcd6137c245c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.087009 15683 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.088495 16427 tablet_bootstrap.cc:492] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:35.089157 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80afbd22fdad4bcdad3768c4a1953f0d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.089210 12718 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.089718 15352 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.090121 16427 tablet_bootstrap.cc:654] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.092368 12437 catalog_manager.cc:5484] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.094879 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.095147 15922 tablet_replica.cc:330] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.095235 16422 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:35.095300 15922 raft_consensus.cc:2229] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.095470 15922 raft_consensus.cc:2258] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.096125 16422 tablet_bootstrap.cc:654] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.096181 16304 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.097824 12978 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.099622 12848 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.100477 12445 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.102694 14957 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.105952 16422 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:35.106089 16422 ts_tablet_manager.cc:1367] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:35.106611 16422 raft_consensus.cc:348] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.106689 16384 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.106864 16422 raft_consensus.cc:374] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.107010 16422 raft_consensus.cc:729] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.107021 16384 tablet_bootstrap.cc:492] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap complete.
I20240501 14:09:35.107192 16384 ts_tablet_manager.cc:1367] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20240501 14:09:35.107221 16422 consensus_queue.cc:260] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.107643 16422 ts_tablet_manager.cc:1398] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.107713 16384 raft_consensus.cc:348] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:35.107857 16422 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:35.107996 16384 raft_consensus.cc:729] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: LEARNER
I20240501 14:09:35.108747 16422 tablet_bootstrap.cc:654] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.108723 12718 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:35.109097 16418 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:35.109361 15922 ts_tablet_manager.cc:1875] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.109738 16367 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:35.110002 16367 tablet_bootstrap.cc:492] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:35.110169 16367 ts_tablet_manager.cc:1367] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.166s user 0.026s sys 0.004s
I20240501 14:09:35.108177 16384 consensus_queue.cc:260] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:35.110497 16416 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:35.110550 16384 ts_tablet_manager.cc:1398] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:35.111047 16367 raft_consensus.cc:348] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:35.111229 16367 raft_consensus.cc:729] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:35.111416 16367 consensus_queue.cc:260] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:35.111764 15922 ts_tablet_manager.cc:1888] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.112592 16367 ts_tablet_manager.cc:1398] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.113191 16418 tablet_bootstrap.cc:654] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.113483 16422 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:35.113564 16422 ts_tablet_manager.cc:1367] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:35.113979 16422 raft_consensus.cc:348] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.114086 16422 raft_consensus.cc:374] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.114116 16422 raft_consensus.cc:729] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.114248 16422 consensus_queue.cc:260] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.114297 16416 tablet_bootstrap.cc:654] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.114567 16422 ts_tablet_manager.cc:1398] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.114534 15242 tablet_copy_service.cc:342] P 66e2dc34330848c3b8c2dd15898d34c8: Request end of tablet copy session 6a3cc0660eed427eaee38164a8c1d4c3-41a1a3342cae4af480f1e65164493219 received from {username='slave'} at 127.6.82.1:43007
I20240501 14:09:35.114622 15242 tablet_copy_service.cc:433] P 66e2dc34330848c3b8c2dd15898d34c8: ending tablet copy session 6a3cc0660eed427eaee38164a8c1d4c3-41a1a3342cae4af480f1e65164493219 on tablet 41a1a3342cae4af480f1e65164493219 with peer 6a3cc0660eed427eaee38164a8c1d4c3
I20240501 14:09:35.114720 16422 tablet_bootstrap.cc:492] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:35.115736 16422 tablet_bootstrap.cc:654] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.115298 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-1a99b39e354144e48eee75f9f49a09fc received from {username='slave'} at 127.6.82.7:51545
I20240501 14:09:35.116115 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-1a99b39e354144e48eee75f9f49a09fc on tablet 1a99b39e354144e48eee75f9f49a09fc with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:35.116175 16419 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:35.116247 16419 ts_tablet_manager.cc:1367] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240501 14:09:35.116354 15922 log.cc:1196] T 80afbd22fdad4bcdad3768c4a1953f0d P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:35.116637 16419 raft_consensus.cc:348] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.116720 16419 raft_consensus.cc:374] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.116758 16419 raft_consensus.cc:729] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.116889 16419 consensus_queue.cc:260] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.116915 12978 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.117172 16419 ts_tablet_manager.cc:1398] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.117391 16305 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.118077 14957 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.118500 16305 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.119486 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 80afbd22fdad4bcdad3768c4a1953f0d (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:35.119709 16422 tablet_bootstrap.cc:492] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:35.119802 16422 ts_tablet_manager.cc:1367] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:35.120288 16422 raft_consensus.cc:348] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.120452 16422 raft_consensus.cc:374] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.120529 16422 raft_consensus.cc:729] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.120728 16422 consensus_queue.cc:260] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.121138 16422 ts_tablet_manager.cc:1398] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:35.121165 16417 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:35.121331 16417 ts_tablet_manager.cc:1367] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20240501 14:09:35.121901 16417 raft_consensus.cc:348] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.123034 16416 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:35.123490 16416 ts_tablet_manager.cc:1367] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:35.123916 16416 raft_consensus.cc:348] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.123989 16416 raft_consensus.cc:374] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.124009 16416 raft_consensus.cc:729] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.124106 16416 consensus_queue.cc:260] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.124356 16416 ts_tablet_manager.cc:1398] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.122097 16417 raft_consensus.cc:374] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.125857 16417 raft_consensus.cc:729] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.126094 16417 consensus_queue.cc:260] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.126907 16417 ts_tablet_manager.cc:1398] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:35.127067 16417 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:35.128470 16417 tablet_bootstrap.cc:654] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.130071 16305 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.130450 16427 tablet_bootstrap.cc:492] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:35.130534 16427 ts_tablet_manager.cc:1367] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.000s
I20240501 14:09:35.130981 16427 raft_consensus.cc:348] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.131090 16427 raft_consensus.cc:374] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.131122 16427 raft_consensus.cc:729] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.131203 12978 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.131458 16427 consensus_queue.cc:260] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.131722 16427 ts_tablet_manager.cc:1398] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.132179 16417 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:35.132261 16417 ts_tablet_manager.cc:1367] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:35.133854 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet abac52c109174021ac11360b0d22ccfa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.134441 16418 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:35.134541 16418 ts_tablet_manager.cc:1367] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 14:09:35.134477 14957 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.135166 16418 raft_consensus.cc:348] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.135283 16418 raft_consensus.cc:374] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.135320 16418 raft_consensus.cc:729] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.135465 16418 consensus_queue.cc:260] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.135821 16418 ts_tablet_manager.cc:1398] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.137302 12440 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.139169 16418 tablet_bootstrap.cc:492] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:35.140031 16418 tablet_bootstrap.cc:654] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.141103 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet abac52c109174021ac11360b0d22ccfa with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:35.141351 16379 tablet_replica.cc:330] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:35.141505 16379 raft_consensus.cc:2229] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.141669 16379 raft_consensus.cc:2258] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.132926 16417 raft_consensus.cc:348] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.143533 16417 raft_consensus.cc:374] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.143625 16417 raft_consensus.cc:729] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.143904 16417 consensus_queue.cc:260] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.145293 16417 ts_tablet_manager.cc:1398] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 14:09:35.145998 16379 ts_tablet_manager.cc:1875] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.147966 15422 raft_consensus.cc:482] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.148074 15422 raft_consensus.cc:504] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.148290 16379 ts_tablet_manager.cc:1888] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.148353 16379 log.cc:1196] T abac52c109174021ac11360b0d22ccfa P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/abac52c109174021ac11360b0d22ccfa
I20240501 14:09:35.148524 15422 leader_election.cc:288] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:35.149092 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:35.149250 15222 raft_consensus.cc:2454] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:35.149586 15023 leader_election.cc:302] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:35.150395 15422 raft_consensus.cc:2781] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.150462 15422 raft_consensus.cc:482] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.150511 15422 raft_consensus.cc:3037] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.150560 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet abac52c109174021ac11360b0d22ccfa (table rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]) successfully deleted
I20240501 14:09:35.150261 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:35.151707 15422 raft_consensus.cc:504] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.152166 15422 leader_election.cc:288] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:35.153004 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:35.153156 15222 raft_consensus.cc:3037] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.153436 16417 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:35.154350 15222 raft_consensus.cc:2454] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:35.154228 16417 tablet_bootstrap.cc:654] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
W20240501 14:09:35.155129 15026 leader_election.cc:341] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:35.155541 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:35.155843 16418 tablet_bootstrap.cc:492] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:35.155938 16418 ts_tablet_manager.cc:1367] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:35.156100 15023 leader_election.cc:302] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
W20240501 14:09:35.156323 15026 leader_election.cc:341] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:35.156348 16418 raft_consensus.cc:348] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.156452 16418 raft_consensus.cc:374] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.156481 16418 raft_consensus.cc:729] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.156498 15422 raft_consensus.cc:2781] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.156580 15422 raft_consensus.cc:686] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:35.156623 16418 consensus_queue.cc:260] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.156754 15422 consensus_queue.cc:237] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.156867 16418 ts_tablet_manager.cc:1398] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.160269 12440 catalog_manager.cc:5484] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.188464 15571 raft_consensus.cc:482] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.188606 15571 raft_consensus.cc:504] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.189249 15571 leader_election.cc:288] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:35.203188 16417 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:35.203305 16417 ts_tablet_manager.cc:1367] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20240501 14:09:35.203816 16417 raft_consensus.cc:348] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.203909 16417 raft_consensus.cc:374] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.203930 16417 raft_consensus.cc:729] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.204037 16417 consensus_queue.cc:260] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.204286 16417 ts_tablet_manager.cc:1398] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.207461 15895 raft_consensus.cc:482] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.207587 15895 raft_consensus.cc:504] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.208115 15895 leader_election.cc:288] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:35.208447 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:35.208567 15352 raft_consensus.cc:2454] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:35.208716 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:35.208869 15222 raft_consensus.cc:2454] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:35.209309 14891 leader_election.cc:302] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:35.209679 15895 raft_consensus.cc:2781] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.209743 15895 raft_consensus.cc:482] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.209790 15895 raft_consensus.cc:3037] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.211628 15895 raft_consensus.cc:504] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.212057 15895 leader_election.cc:288] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:35.212553 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:35.212651 15352 raft_consensus.cc:3037] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.213539 15352 raft_consensus.cc:2454] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:35.214139 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:35.214257 15222 raft_consensus.cc:3037] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.214892 14894 leader_election.cc:302] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:35.215492 15222 raft_consensus.cc:2454] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:35.215924 15895 raft_consensus.cc:2781] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.216641 16391 raft_consensus.cc:482] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.216753 16391 raft_consensus.cc:504] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.217114 16391 leader_election.cc:288] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:35.217121 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:35.217473 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089b7a68a30e48f5bbcd6137c245c6a2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:35.217576 14958 raft_consensus.cc:2454] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:35.217635 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089b7a68a30e48f5bbcd6137c245c6a2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:35.216317 15895 raft_consensus.cc:686] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:35.217758 15089 raft_consensus.cc:2454] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:35.217907 15895 consensus_queue.cc:237] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.217995 12914 leader_election.cc:302] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:35.218267 16391 raft_consensus.cc:2781] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.218322 16391 raft_consensus.cc:482] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.218359 16391 raft_consensus.cc:3037] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.219388 16391 raft_consensus.cc:504] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.219703 16391 leader_election.cc:288] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:35.220153 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089b7a68a30e48f5bbcd6137c245c6a2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:35.220254 14957 raft_consensus.cc:3037] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.220837 12445 catalog_manager.cc:5484] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.221189 14957 raft_consensus.cc:2454] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:35.221896 12914 leader_election.cc:302] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:35.222097 16391 raft_consensus.cc:2781] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.222162 16391 raft_consensus.cc:686] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:35.222266 16391 consensus_queue.cc:237] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:35.222787 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089b7a68a30e48f5bbcd6137c245c6a2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:35.222915 15089 raft_consensus.cc:3037] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.223867 15089 raft_consensus.cc:2454] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:35.224599 12440 catalog_manager.cc:5484] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.225083 15571 raft_consensus.cc:482] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.225168 15571 raft_consensus.cc:504] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.225533 15571 leader_election.cc:288] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:35.226171 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfce6c5fb6344a878935b322b5d8e2d0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:35.226296 12718 raft_consensus.cc:2454] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:35.226914 15286 leader_election.cc:302] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:35.230309 15571 raft_consensus.cc:2781] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.230360 15571 raft_consensus.cc:482] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.230401 15571 raft_consensus.cc:3037] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.231314 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfce6c5fb6344a878935b322b5d8e2d0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:35.231424 15571 raft_consensus.cc:504] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.231460 15089 raft_consensus.cc:2454] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:35.231765 15571 leader_election.cc:288] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:35.232738 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfce6c5fb6344a878935b322b5d8e2d0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:35.232848 12717 raft_consensus.cc:3037] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.233881 12717 raft_consensus.cc:2454] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:35.234915 15286 leader_election.cc:302] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:35.235152 16474 raft_consensus.cc:2781] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.235515 16474 raft_consensus.cc:686] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:35.235754 16474 consensus_queue.cc:237] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.236426 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:35.236624 15222 raft_consensus.cc:2379] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f23f426391724d6688aafffe7ee017d8 in current term 1: Already voted for candidate 8a13c028ba8a4a659b74af6c265df95e in this term.
I20240501 14:09:35.238207 15286 leader_election.cc:302] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23f426391724d6688aafffe7ee017d8; no voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:35.238193 12440 catalog_manager.cc:5484] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.238646 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfce6c5fb6344a878935b322b5d8e2d0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:35.238785 15089 raft_consensus.cc:3037] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.239733 15089 raft_consensus.cc:2454] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:35.240851 15571 raft_consensus.cc:2726] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:35.249032 16107 raft_consensus.cc:1050] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:35.249445 16107 consensus_queue.cc:237] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:35.250767 15089 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.251224 14452 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.253159 12848 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.253175 12978 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.254149 14452 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.254658 14452 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.260325 16305 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.262343 15089 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.262555 12978 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.265105 12848 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.265938 12442 catalog_manager.cc:5484] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.271538 12718 consensus_queue.cc:237] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:35.272922 12848 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.273954 15089 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.274304 14452 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.275631 14452 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.276494 14452 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.276856 12848 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.278051 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e13e55ae52a949bc893de94082130157 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.279357 12439 catalog_manager.cc:5484] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.282152 15089 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.282378 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet e13e55ae52a949bc893de94082130157 with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:35.282622 15925 tablet_replica.cc:330] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:35.282780 15925 raft_consensus.cc:2229] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.282964 15925 raft_consensus.cc:2258] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.284559 15925 ts_tablet_manager.cc:1875] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.286921 15925 ts_tablet_manager.cc:1888] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.287000 15925 log.cc:1196] T e13e55ae52a949bc893de94082130157 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/e13e55ae52a949bc893de94082130157
I20240501 14:09:35.290189 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet e13e55ae52a949bc893de94082130157 (table rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]) successfully deleted
I20240501 14:09:35.294687 12829 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:35.305150 14938 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:35.306286 15200 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.311400 12696 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 14:09:35.314993 15070 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.327610 12958 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 14:09:35.330660 15331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:35.343497 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:35.350708 14957 raft_consensus.cc:1261] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.351016 15089 raft_consensus.cc:1261] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.351469 16389 consensus_queue.cc:1035] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.352759 16389 consensus_queue.cc:1035] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.360373 12440 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]). Expected version 1 got 0
I20240501 14:09:35.362102 12440 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]). Expected version 1 got 0
I20240501 14:09:35.364022 12565 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:35.392411 15683 raft_consensus.cc:1050] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:35.392951 15683 consensus_queue.cc:237] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:35.394675 12585 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.395440 16360 raft_consensus.cc:1050] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:35.396044 16360 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:35.397979 14674 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.398136 12718 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.399389 12978 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.400765 15878 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.401226 12718 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.402346 16375 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.403982 15352 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.404039 15088 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.404445 15878 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.404742 15878 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.413157 16508 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:35.415362 15222 raft_consensus.cc:1203] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:35.415730 16391 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.427415 12440 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]). Expected version 1 got 0
I20240501 14:09:35.427922 12439 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]). Expected version 1 got 0
I20240501 14:09:35.428390 12442 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]). Expected version 1 got 0
I20240501 14:09:35.429677 16375 raft_consensus.cc:1050] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:35.430047 16375 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:35.431766 12718 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.431829 12442 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:09:35.431946 12442 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 14:09:35.432358 15683 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.432866 12848 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.433615 14958 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.433575 15683 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.435999 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet afa5ebe09c99435fb69ceb527ab4734a
I20240501 14:09:35.436208 15683 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.436322 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 25552a578587473fa9f22ba47b425bc1
I20240501 14:09:35.436558 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 16daa4d3a11e47a9ae9764e70a93fa0b
I20240501 14:09:35.437337 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5ebe09c99435fb69ceb527ab4734a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.437402 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 25552a578587473fa9f22ba47b425bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:35.437518 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 25552a578587473fa9f22ba47b425bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.437569 16379 tablet_replica.cc:330] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:35.437637 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 16daa4d3a11e47a9ae9764e70a93fa0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.437714 16379 raft_consensus.cc:2229] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.437774 15922 tablet_replica.cc:330] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.437846 16379 raft_consensus.cc:2258] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.437911 15922 raft_consensus.cc:2229] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.438127 15922 raft_consensus.cc:2258] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.438136 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 16daa4d3a11e47a9ae9764e70a93fa0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:35.438288 15925 tablet_replica.cc:330] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:35.438354 15070 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5ebe09c99435fb69ceb527ab4734a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:35.438408 15925 raft_consensus.cc:2229] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.438448 16379 ts_tablet_manager.cc:1875] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.438548 15925 raft_consensus.cc:2258] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.438578 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 25552a578587473fa9f22ba47b425bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:35.438575 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3] alter complete (version 1)
I20240501 14:09:35.438925 16106 tablet_replica.cc:330] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:35.439042 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5ebe09c99435fb69ceb527ab4734a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.439107 16106 raft_consensus.cc:2229] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.439195 15923 tablet_replica.cc:330] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.439285 16106 raft_consensus.cc:2258] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.439306 15923 raft_consensus.cc:2229] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.439478 15923 raft_consensus.cc:2258] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.439486 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 16daa4d3a11e47a9ae9764e70a93fa0b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.440209 15923 ts_tablet_manager.cc:1875] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.440539 15922 ts_tablet_manager.cc:1875] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.440752 16379 ts_tablet_manager.cc:1888] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.440815 16379 log.cc:1196] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/25552a578587473fa9f22ba47b425bc1
I20240501 14:09:35.441064 16379 ts_tablet_manager.cc:1909] T 25552a578587473fa9f22ba47b425bc1 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:35.441807 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 25552a578587473fa9f22ba47b425bc1 (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.442386 15923 ts_tablet_manager.cc:1888] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:35.442445 15923 log.cc:1196] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/afa5ebe09c99435fb69ceb527ab4734a
I20240501 14:09:35.443194 15925 ts_tablet_manager.cc:1875] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.443346 16106 ts_tablet_manager.cc:1875] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.445228 15925 ts_tablet_manager.cc:1888] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.445298 15925 log.cc:1196] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/16daa4d3a11e47a9ae9764e70a93fa0b
I20240501 14:09:35.445554 15925 ts_tablet_manager.cc:1909] T 16daa4d3a11e47a9ae9764e70a93fa0b P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:35.445938 16106 ts_tablet_manager.cc:1888] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:35.446024 16106 log.cc:1196] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/afa5ebe09c99435fb69ceb527ab4734a
I20240501 14:09:35.446130 15925 tablet_replica.cc:330] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:35.446231 15925 raft_consensus.cc:2229] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.446373 15925 raft_consensus.cc:2258] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.446381 16106 ts_tablet_manager.cc:1909] T afa5ebe09c99435fb69ceb527ab4734a P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:35.448737 15922 ts_tablet_manager.cc:1888] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:35.448896 15922 log.cc:1196] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/afa5ebe09c99435fb69ceb527ab4734a
I20240501 14:09:35.449385 15922 ts_tablet_manager.cc:1909] T afa5ebe09c99435fb69ceb527ab4734a P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:35.449561 15925 ts_tablet_manager.cc:1875] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.449718 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 16daa4d3a11e47a9ae9764e70a93fa0b (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.442708 15923 ts_tablet_manager.cc:1909] T afa5ebe09c99435fb69ceb527ab4734a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:35.450594 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet afa5ebe09c99435fb69ceb527ab4734a (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.451421 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet afa5ebe09c99435fb69ceb527ab4734a (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.451818 15922 tablet_replica.cc:330] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.451916 15923 tablet_replica.cc:330] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.452066 15922 raft_consensus.cc:2229] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.452170 15923 raft_consensus.cc:2229] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.452248 15922 raft_consensus.cc:2258] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.452874 15922 ts_tablet_manager.cc:1875] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.453928 15923 raft_consensus.cc:2258] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.454566 15923 ts_tablet_manager.cc:1875] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.455439 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet afa5ebe09c99435fb69ceb527ab4734a (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.455832 15922 ts_tablet_manager.cc:1888] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.456000 15922 log.cc:1196] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/25552a578587473fa9f22ba47b425bc1
I20240501 14:09:35.456352 15922 ts_tablet_manager.cc:1909] T 25552a578587473fa9f22ba47b425bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:35.457087 15922 tablet_replica.cc:330] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.457187 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 25552a578587473fa9f22ba47b425bc1 (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.458169 15925 ts_tablet_manager.cc:1888] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.458822 15925 log.cc:1196] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/25552a578587473fa9f22ba47b425bc1
I20240501 14:09:35.459184 15925 ts_tablet_manager.cc:1909] T 25552a578587473fa9f22ba47b425bc1 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:35.460192 15923 ts_tablet_manager.cc:1888] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.460321 15923 log.cc:1196] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/16daa4d3a11e47a9ae9764e70a93fa0b
I20240501 14:09:35.460660 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 25552a578587473fa9f22ba47b425bc1 (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.461154 15923 ts_tablet_manager.cc:1909] T 16daa4d3a11e47a9ae9764e70a93fa0b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:35.461974 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 16daa4d3a11e47a9ae9764e70a93fa0b (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.462957 16360 raft_consensus.cc:1050] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:35.462998 16508 tablet.cc:1620] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:35.463485 16360 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:35.464952 12978 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.465333 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.468093 15878 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.469156 12718 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.469597 16502 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.470281 15683 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.471189 12716 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.472432 12439 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.472798 15353 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.473549 12980 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.474108 12716 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.476562 14957 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.476786 15684 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.479116 12442 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.480881 12848 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.481688 12716 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.483011 15222 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.483680 16502 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.479118 12585 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.491065 15922 raft_consensus.cc:2229] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.491271 15922 raft_consensus.cc:2258] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.491860 15922 ts_tablet_manager.cc:1875] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.492519 15353 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.493657 16501 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.495699 12439 catalog_manager.cc:5484] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.495672 12978 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.497074 12848 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:35.498567 16511 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:35.499279 16509 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:35.499359 12718 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.499646 16509 tablet.cc:1620] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:35.499747 12978 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.500269 16502 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.501386 15352 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.503710 16360 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.504117 12978 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.504176 12439 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.506228 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d64234effb3446bba1ff82bdf37acaa7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.506982 15353 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.509419 12848 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:35.510006 16502 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.510984 15351 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.511883 15353 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.512853 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.513105 12980 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:35.513557 12979 consensus_queue.cc:237] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:35.514434 12440 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.514804 14957 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.515635 16391 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.515945 15222 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.516631 12718 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.517123 12848 raft_consensus.cc:1261] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.517446 15683 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.517765 15683 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.517830 16390 consensus_queue.cc:1035] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.518303 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet d64234effb3446bba1ff82bdf37acaa7 with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.519757 15923 tablet_replica.cc:330] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.519919 15923 raft_consensus.cc:2229] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.519845 16391 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.520085 15923 raft_consensus.cc:2258] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.520474 12978 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.520929 16360 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.521121 12848 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.521842 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8147fae0de98429ab6554c7e7f5bf8ee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.522377 15222 raft_consensus.cc:2932] T 8147fae0de98429ab6554c7e7f5bf8ee P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.525740 16390 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.526119 12439 catalog_manager.cc:5484] T 8147fae0de98429ab6554c7e7f5bf8ee P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.527083 14957 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.527554 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25b5c40793b348bbb0e5251c8d553e30 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.527629 15923 ts_tablet_manager.cc:1875] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.528172 12718 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.529160 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 8147fae0de98429ab6554c7e7f5bf8ee with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.530014 12441 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:35.530439 16360 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.532313 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.532332 12979 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.532629 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b5c40793b348bbb0e5251c8d553e30 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:35.532816 16379 tablet_replica.cc:330] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:35.532953 16379 raft_consensus.cc:2229] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.533092 16379 raft_consensus.cc:2258] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.533852 16379 ts_tablet_manager.cc:1875] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.535943 12440 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.536207 16379 ts_tablet_manager.cc:1888] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.536280 16379 log.cc:1196] T 25b5c40793b348bbb0e5251c8d553e30 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/25b5c40793b348bbb0e5251c8d553e30
I20240501 14:09:35.537428 16511 tablet.cc:1620] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:35.538978 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 25b5c40793b348bbb0e5251c8d553e30 (table rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]) successfully deleted
I20240501 14:09:35.539062 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.539465 15353 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:35.539608 15089 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.542405 12585 raft_consensus.cc:1203] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:35.543305 15923 ts_tablet_manager.cc:1888] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.543395 15923 log.cc:1196] T d64234effb3446bba1ff82bdf37acaa7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/d64234effb3446bba1ff82bdf37acaa7
I20240501 14:09:35.544241 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet d64234effb3446bba1ff82bdf37acaa7 (table rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]) successfully deleted
I20240501 14:09:35.544304 15922 ts_tablet_manager.cc:1888] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.544392 15922 log.cc:1196] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/16daa4d3a11e47a9ae9764e70a93fa0b
I20240501 14:09:35.544685 15923 tablet_replica.cc:330] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.544840 15923 raft_consensus.cc:2229] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.546856 12978 consensus_queue.cc:237] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:35.548744 12442 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:35.549366 15089 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.549896 16391 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.549998 12718 raft_consensus.cc:1261] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.545020 15923 raft_consensus.cc:2258] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.551039 16391 consensus_queue.cc:1035] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.551612 15923 ts_tablet_manager.cc:1875] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.552755 16391 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.552871 15922 ts_tablet_manager.cc:1909] T 16daa4d3a11e47a9ae9764e70a93fa0b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:35.553776 15923 ts_tablet_manager.cc:1888] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.553856 15923 log.cc:1196] T 8147fae0de98429ab6554c7e7f5bf8ee P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/8147fae0de98429ab6554c7e7f5bf8ee
I20240501 14:09:35.553723 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 16daa4d3a11e47a9ae9764e70a93fa0b (table rebalance_test_table_10 [id=efedc2e3f1ca4c2581dbe60f1f8bfab5]) successfully deleted
I20240501 14:09:35.554333 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 8147fae0de98429ab6554c7e7f5bf8ee (table rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]) successfully deleted
I20240501 14:09:35.555136 15923 tablet_replica.cc:330] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.555299 15923 raft_consensus.cc:2229] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.555451 15923 raft_consensus.cc:2258] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.555696 12718 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.556396 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d2631324f66d454697b895ecdd227276 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.558941 15089 raft_consensus.cc:2932] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.559087 12442 catalog_manager.cc:5484] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:35.562084 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet d2631324f66d454697b895ecdd227276 with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.562417 15922 tablet_replica.cc:330] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.562659 15922 raft_consensus.cc:2229] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.563206 16508 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:35.563428 16511 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:35.563534 16508 tablet.cc:1620] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:35.563874 16511 tablet.cc:1620] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:35.562924 15922 raft_consensus.cc:2258] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.565200 15922 ts_tablet_manager.cc:1875] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.567163 16509 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:35.567502 16509 tablet.cc:1620] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:35.570431 15922 ts_tablet_manager.cc:1888] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.571075 15923 ts_tablet_manager.cc:1875] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.571758 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3] alter complete (version 2)
I20240501 14:09:35.574295 15923 ts_tablet_manager.cc:1888] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.574440 15923 log.cc:1196] T bd5489eab5b04751bc8b3e5fb9e8cf73 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd5489eab5b04751bc8b3e5fb9e8cf73
I20240501 14:09:35.574615 15922 log.cc:1196] T d2631324f66d454697b895ecdd227276 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/d2631324f66d454697b895ecdd227276
I20240501 14:09:35.575109 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet bd5489eab5b04751bc8b3e5fb9e8cf73 (table rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]) successfully deleted
I20240501 14:09:35.575805 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet d2631324f66d454697b895ecdd227276 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:35.589075 15895 raft_consensus.cc:1050] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e: attempting to promote NON_VOTER a6e94562bee149c2ace25672f0f9bf50 to VOTER
I20240501 14:09:35.589501 15895 consensus_queue.cc:237] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:35.590941 12848 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:35.591126 15222 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:35.591646 12978 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:35.591923 15895 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:35.592082 15353 raft_consensus.cc:3037] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.592080 15421 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:35.592442 15421 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:35.594091 15422 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.594556 15421 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.597687 15222 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.599169 12441 catalog_manager.cc:5484] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.599319 12978 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.599570 12848 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.601580 15425 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.606885 14957 consensus_queue.cc:237] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:35.608426 12978 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.608506 12848 raft_consensus.cc:1261] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.608747 16522 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.609272 15421 consensus_queue.cc:1035] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.610332 16523 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.610620 12978 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.611837 12848 raft_consensus.cc:2932] T 5dd29fa5dc2b4408878ceb3c12a57098 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.614090 12442 catalog_manager.cc:5484] T 5dd29fa5dc2b4408878ceb3c12a57098 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:35.614683 12430 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5dd29fa5dc2b4408878ceb3c12a57098 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.616806 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dd29fa5dc2b4408878ceb3c12a57098 with delete_type TABLET_DATA_TOMBSTONED (TS 66e2dc34330848c3b8c2dd15898d34c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:35.617303 16536 tablet_replica.cc:330] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:35.617455 16536 raft_consensus.cc:2229] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.617621 16536 raft_consensus.cc:2258] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.618888 16536 ts_tablet_manager.cc:1875] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.621328 16536 ts_tablet_manager.cc:1888] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:35.621402 16536 log.cc:1196] T 5dd29fa5dc2b4408878ceb3c12a57098 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/5dd29fa5dc2b4408878ceb3c12a57098
I20240501 14:09:35.621965 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 5dd29fa5dc2b4408878ceb3c12a57098 (table rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]) successfully deleted
I20240501 14:09:35.640725 16475 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.645213 16474 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.676539 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:35.682603 16508 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:35.683008 16511 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:35.686969 16509 tablet.cc:1722] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:35.689682 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3] alter complete (version 3)
I20240501 14:09:35.733371 12848 consensus_queue.cc:237] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:35.735074 12585 raft_consensus.cc:1261] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.735183 12718 raft_consensus.cc:1261] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.735453 16501 consensus_queue.cc:1035] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.735612 16360 consensus_queue.cc:1035] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.737761 16360 raft_consensus.cc:2932] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:35.738140 12718 raft_consensus.cc:2932] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
W20240501 14:09:35.738792 12783 consensus_peers.cc:485] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a74c6b1b2dc4793a8d7ed4c41de013d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:35.739463 12585 raft_consensus.cc:2932] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:35.741665 12442 catalog_manager.cc:5484] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.749306 14957 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:35.750656 15088 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.750656 15222 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:35.750924 14894 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df57dee35f924801a23bd9b5c547a55f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:35.751035 15421 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:35.751206 16523 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:35.752808 16522 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.754446 15222 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.754441 15088 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.755316 12440 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.764519 12718 consensus_queue.cc:237] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:35.765892 12848 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.765888 12585 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.766281 16305 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.766471 15980 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:35.766678 12653 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 515d8f8e9d744aa4b2c8a4090c2f72e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:35.768198 16304 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:35.768697 12585 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:35.768675 12848 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:35.771610 12441 catalog_manager.cc:5484] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:35.774523 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:35.777086 16522 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.781153 15421 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:35.781654 12978 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:35.782574 14938 tablet_service.cc:1467] Processing CreateTablet for tablet ee9a1e9b13f546cc81aae6d729c8974f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.782927 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9a1e9b13f546cc81aae6d729c8974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.783368 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 5b037b6095a644fda854cf0365b1a7df (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.783435 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 3d6a7ed55367493e9edd0461d6da6313 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.783648 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b037b6095a644fda854cf0365b1a7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.783807 15202 tablet_service.cc:1467] Processing CreateTablet for tablet ee9a1e9b13f546cc81aae6d729c8974f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.784022 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9a1e9b13f546cc81aae6d729c8974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.784027 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 3d6a7ed55367493e9edd0461d6da6313 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.784267 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6a7ed55367493e9edd0461d6da6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.785388 12698 tablet_service.cc:1467] Processing CreateTablet for tablet ee9a1e9b13f546cc81aae6d729c8974f (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.785670 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9a1e9b13f546cc81aae6d729c8974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.786027 15088 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.786639 14957 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:35.786633 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 5b037b6095a644fda854cf0365b1a7df (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.786882 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b037b6095a644fda854cf0365b1a7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.787094 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6a7ed55367493e9edd0461d6da6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.787312 16653 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:35.788169 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 5b037b6095a644fda854cf0365b1a7df (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.788429 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b037b6095a644fda854cf0365b1a7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.788601 16653 tablet_bootstrap.cc:654] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.788918 16375 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:35.789311 16651 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:35.789369 16652 tablet_bootstrap.cc:492] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:35.790302 16651 tablet_bootstrap.cc:654] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.791709 16391 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:35.792081 16651 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:35.792212 16651 ts_tablet_manager.cc:1367] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:35.792954 15683 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.793300 16652 tablet_bootstrap.cc:654] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.793956 16655 tablet_bootstrap.cc:492] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:35.793947 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 3d6a7ed55367493e9edd0461d6da6313 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:35.794216 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6a7ed55367493e9edd0461d6da6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.794226 12441 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:35.794332 12441 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:35.795112 16655 tablet_bootstrap.cc:654] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.796077 16655 tablet_bootstrap.cc:492] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:35.796134 16655 ts_tablet_manager.cc:1367] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:35.796487 16661 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:35.796530 16655 raft_consensus.cc:348] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:35.796658 16655 raft_consensus.cc:374] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.796689 16655 raft_consensus.cc:729] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.796830 16655 consensus_queue.cc:260] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:35.796878 14957 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.797107 16655 ts_tablet_manager.cc:1398] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.797375 16661 tablet_bootstrap.cc:654] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.798266 16652 tablet_bootstrap.cc:492] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:35.798352 16652 ts_tablet_manager.cc:1367] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:35.798569 16661 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:35.798640 16661 ts_tablet_manager.cc:1367] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.798871 16652 raft_consensus.cc:348] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:35.798967 16652 raft_consensus.cc:374] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.799037 16652 raft_consensus.cc:729] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.799034 16661 raft_consensus.cc:348] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.799126 16661 raft_consensus.cc:374] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.799157 16661 raft_consensus.cc:729] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.799193 16652 consensus_queue.cc:260] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:35.799247 16661 consensus_queue.cc:260] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.799500 16661 ts_tablet_manager.cc:1398] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.800493 12442 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.801393 15088 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:35.801842 16652 ts_tablet_manager.cc:1398] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20240501 14:09:35.802197 12916 consensus_peers.cc:485] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 188ce319682147ddaad30aae7232a311. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:35.802456 16652 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:35.804975 16654 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:35.805002 16652 tablet_bootstrap.cc:654] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.806252 16652 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:35.806355 16652 ts_tablet_manager.cc:1367] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:35.806908 16652 raft_consensus.cc:348] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.806985 16654 tablet_bootstrap.cc:654] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.807015 16652 raft_consensus.cc:374] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.807047 16652 raft_consensus.cc:729] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.807165 16652 consensus_queue.cc:260] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.808185 16653 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:35.808454 16652 ts_tablet_manager.cc:1398] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:35.808727 12441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 089b7a68a30e48f5bbcd6137c245c6a2
I20240501 14:09:35.808748 16653 ts_tablet_manager.cc:1367] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:35.809212 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 089b7a68a30e48f5bbcd6137c245c6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:35.809460 16653 raft_consensus.cc:348] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.809579 16653 raft_consensus.cc:374] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.809609 16653 raft_consensus.cc:729] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.809823 16653 consensus_queue.cc:260] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.809491 16652 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:35.810196 16653 ts_tablet_manager.cc:1398] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.810424 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 089b7a68a30e48f5bbcd6137c245c6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:35.810590 16106 tablet_replica.cc:330] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:35.810575 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 089b7a68a30e48f5bbcd6137c245c6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:35.810691 16106 raft_consensus.cc:2229] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.810829 16106 raft_consensus.cc:2258] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.811008 15925 tablet_replica.cc:330] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:35.811128 15925 raft_consensus.cc:2229] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.811317 15925 raft_consensus.cc:2258] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.811705 16652 tablet_bootstrap.cc:654] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.811939 16106 ts_tablet_manager.cc:1875] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.813138 16652 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:35.813231 16652 ts_tablet_manager.cc:1367] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:35.813529 12848 consensus_queue.cc:237] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:35.813612 16652 raft_consensus.cc:348] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.813694 16652 raft_consensus.cc:374] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.813722 16652 raft_consensus.cc:729] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.813827 16652 consensus_queue.cc:260] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:35.814085 16652 ts_tablet_manager.cc:1398] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.814178 16106 ts_tablet_manager.cc:1888] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.814241 16106 log.cc:1196] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/089b7a68a30e48f5bbcd6137c245c6a2
I20240501 14:09:35.814513 16106 ts_tablet_manager.cc:1909] T 089b7a68a30e48f5bbcd6137c245c6a2 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:35.815125 15925 ts_tablet_manager.cc:1875] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.816073 12978 raft_consensus.cc:1261] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.816520 16500 consensus_queue.cc:1035] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.817126 15925 ts_tablet_manager.cc:1888] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.817183 15925 log.cc:1196] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/089b7a68a30e48f5bbcd6137c245c6a2
I20240501 14:09:35.817414 15925 ts_tablet_manager.cc:1909] T 089b7a68a30e48f5bbcd6137c245c6a2 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:35.817917 12585 raft_consensus.cc:1261] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.818298 16500 raft_consensus.cc:2932] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
W20240501 14:09:35.819295 12785 consensus_peers.cc:485] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9f2c6558161436683f50486e437dc28. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:35.819590 16651 raft_consensus.cc:348] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.819715 16651 raft_consensus.cc:374] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.819741 16651 raft_consensus.cc:729] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.819873 16651 consensus_queue.cc:260] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.820143 16651 ts_tablet_manager.cc:1398] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.028s user 0.000s sys 0.000s
I20240501 14:09:35.821063 16360 consensus_queue.cc:1035] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.821009 12978 raft_consensus.cc:2932] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:35.821292 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table_new_name [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]) successfully deleted
I20240501 14:09:35.821205 12439 catalog_manager.cc:5484] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.822430 12585 raft_consensus.cc:2932] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:35.823850 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table_new_name [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]) successfully deleted
I20240501 14:09:35.810427 16653 tablet_bootstrap.cc:492] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:35.825493 16653 tablet_bootstrap.cc:654] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.826666 16653 tablet_bootstrap.cc:492] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:35.826774 16653 ts_tablet_manager.cc:1367] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:35.827225 16653 raft_consensus.cc:348] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:35.827355 16653 raft_consensus.cc:374] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.827401 16653 raft_consensus.cc:729] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.827612 16653 consensus_queue.cc:260] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:35.827934 16654 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:35.827950 16653 ts_tablet_manager.cc:1398] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.828044 16654 ts_tablet_manager.cc:1367] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:35.828459 16654 raft_consensus.cc:348] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.828735 16654 raft_consensus.cc:374] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.829782 16654 raft_consensus.cc:729] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.830299 16654 consensus_queue.cc:260] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.831221 16654 ts_tablet_manager.cc:1398] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:35.833911 16670 tablet_replica.cc:330] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:35.834048 16670 raft_consensus.cc:2229] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.834182 16670 raft_consensus.cc:2258] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.834851 16670 ts_tablet_manager.cc:1875] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.838596 16670 ts_tablet_manager.cc:1888] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.838685 16670 log.cc:1196] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/089b7a68a30e48f5bbcd6137c245c6a2
I20240501 14:09:35.838959 16670 ts_tablet_manager.cc:1909] T 089b7a68a30e48f5bbcd6137c245c6a2 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:35.839572 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 089b7a68a30e48f5bbcd6137c245c6a2 (table rebalancer_dynamic_table_new_name [id=8d41d1ec7b124b8d86f9abc4a1b2ccb3]) successfully deleted
I20240501 14:09:35.865479 16290 raft_consensus.cc:1050] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:35.865916 16290 consensus_queue.cc:237] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:35.867197 12978 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.867715 12718 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.868103 16366 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.868904 16366 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.868956 15222 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:35.869568 16366 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:35.871508 16366 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.873699 12978 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.874205 12440 catalog_manager.cc:5484] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.877904 15222 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.878159 12718 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.882201 12585 consensus_queue.cc:237] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:35.883497 12718 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.883523 15222 raft_consensus.cc:1261] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:35.883858 16366 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.884168 16366 consensus_queue.cc:1035] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:35.885767 16290 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.887249 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a99b39e354144e48eee75f9f49a09fc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.888193 12718 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.888762 12442 catalog_manager.cc:5484] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.889683 15222 raft_consensus.cc:2932] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:35.893046 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99b39e354144e48eee75f9f49a09fc with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:35.893312 15925 tablet_replica.cc:330] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:35.893465 15925 raft_consensus.cc:2229] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.893628 15925 raft_consensus.cc:2258] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.894667 15925 ts_tablet_manager.cc:1875] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.896932 15925 ts_tablet_manager.cc:1888] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:35.897509 15925 log.cc:1196] T 1a99b39e354144e48eee75f9f49a09fc P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/1a99b39e354144e48eee75f9f49a09fc
I20240501 14:09:35.898021 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 1a99b39e354144e48eee75f9f49a09fc (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:35.910126 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:35.915796 15980 raft_consensus.cc:482] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.915858 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5f1e720ec242c094590df7a6987918 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]), partition=
I20240501 14:09:35.915922 15980 raft_consensus.cc:504] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.916162 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5f1e720ec242c094590df7a6987918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.916391 15980 leader_election.cc:288] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:35.916697 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5f1e720ec242c094590df7a6987918 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]), partition=
I20240501 14:09:35.916911 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5f1e720ec242c094590df7a6987918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.917335 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9a1e9b13f546cc81aae6d729c8974f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:35.917472 14957 raft_consensus.cc:2454] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:35.917783 12652 leader_election.cc:302] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:35.917943 15980 raft_consensus.cc:2781] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.917999 15980 raft_consensus.cc:482] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.918042 15980 raft_consensus.cc:3037] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.918411 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5f1e720ec242c094590df7a6987918 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]), partition=
I20240501 14:09:35.918660 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e5f1e720ec242c094590df7a6987918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:35.919034 16661 tablet_bootstrap.cc:492] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:35.919140 15980 raft_consensus.cc:504] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.919507 15980 leader_election.cc:288] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:35.919870 16661 tablet_bootstrap.cc:654] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.919925 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9a1e9b13f546cc81aae6d729c8974f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:35.920027 14957 raft_consensus.cc:3037] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.920840 16652 tablet_bootstrap.cc:492] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:35.920886 14957 raft_consensus.cc:2454] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:35.921427 12652 leader_election.cc:302] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:35.921726 16652 tablet_bootstrap.cc:654] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.921850 15980 raft_consensus.cc:2781] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.921947 15980 raft_consensus.cc:686] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:35.922098 15980 consensus_queue.cc:237] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:35.923141 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9a1e9b13f546cc81aae6d729c8974f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:35.923265 15222 raft_consensus.cc:2454] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:35.923485 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9a1e9b13f546cc81aae6d729c8974f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:35.923570 15221 raft_consensus.cc:3037] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.924470 15221 raft_consensus.cc:2454] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:35.925000 12438 catalog_manager.cc:5484] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.925668 16651 tablet_bootstrap.cc:492] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:35.926573 16651 tablet_bootstrap.cc:654] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:35.930310 16651 tablet_bootstrap.cc:492] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:35.930399 16651 ts_tablet_manager.cc:1367] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:35.930869 16651 raft_consensus.cc:348] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.930984 16651 raft_consensus.cc:374] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.931021 16651 raft_consensus.cc:729] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.931143 16651 consensus_queue.cc:260] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.931409 16651 ts_tablet_manager.cc:1398] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:35.940361 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:09:35.940510 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 14:09:35.942485 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet 8a74c6b1b2dc4793a8d7ed4c41de013d
I20240501 14:09:35.942885 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 6305f3f7cb8846bf843d2d753fd1ae07
I20240501 14:09:35.943112 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet bfc871f734904165ab772321b04a65a4
I20240501 14:09:35.943701 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.943873 15923 tablet_replica.cc:330] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.943951 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:35.943992 15923 raft_consensus.cc:2229] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.944152 15923 raft_consensus.cc:2258] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.944176 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc871f734904165ab772321b04a65a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.944324 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:35.944320 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.944476 16106 tablet_replica.cc:330] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:35.944481 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 6305f3f7cb8846bf843d2d753fd1ae07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:35.944568 16106 raft_consensus.cc:2229] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.944658 16106 raft_consensus.cc:2258] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.945161 16106 ts_tablet_manager.cc:1875] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.945297 16661 tablet_bootstrap.cc:492] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:35.945369 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:35.945394 16661 ts_tablet_manager.cc:1367] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
W20240501 14:09:35.945804 12430 catalog_manager.cc:4797] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): delete failed for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d because the tablet was not found. No further retry: Not found: Tablet not found: 8a74c6b1b2dc4793a8d7ed4c41de013d
I20240501 14:09:35.945852 16661 raft_consensus.cc:348] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.945951 16661 raft_consensus.cc:374] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.945981 16661 raft_consensus.cc:729] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.946102 16661 consensus_queue.cc:260] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.946388 16661 ts_tablet_manager.cc:1398] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.946605 16652 tablet_bootstrap.cc:492] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:35.946689 16652 ts_tablet_manager.cc:1367] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20240501 14:09:35.947062 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc871f734904165ab772321b04a65a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:35.947289 15922 tablet_replica.cc:330] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.947244 16652 raft_consensus.cc:348] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.947368 16652 raft_consensus.cc:374] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:35.947371 15923 ts_tablet_manager.cc:1875] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.947403 16652 raft_consensus.cc:729] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:35.947419 15922 raft_consensus.cc:2229] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.947575 15922 raft_consensus.cc:2258] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.947522 16652 consensus_queue.cc:260] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.947824 16652 ts_tablet_manager.cc:1398] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:35.948078 15922 ts_tablet_manager.cc:1875] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.949586 15923 ts_tablet_manager.cc:1888] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.949653 15923 log.cc:1196] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a74c6b1b2dc4793a8d7ed4c41de013d
I20240501 14:09:35.949934 15923 ts_tablet_manager.cc:1909] T 8a74c6b1b2dc4793a8d7ed4c41de013d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:35.950284 15922 ts_tablet_manager.cc:1888] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.950348 15922 log.cc:1196] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/bfc871f734904165ab772321b04a65a4
I20240501 14:09:35.950601 15922 ts_tablet_manager.cc:1909] T bfc871f734904165ab772321b04a65a4 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:35.951308 15925 tablet_replica.cc:330] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:35.951427 15925 raft_consensus.cc:2229] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.951550 15925 raft_consensus.cc:2258] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.951690 15923 tablet_replica.cc:330] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:35.951807 15923 raft_consensus.cc:2229] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.951974 15925 ts_tablet_manager.cc:1875] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.951995 15923 raft_consensus.cc:2258] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.952466 15923 ts_tablet_manager.cc:1875] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.953624 15925 ts_tablet_manager.cc:1888] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.953673 15925 log.cc:1196] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/bfc871f734904165ab772321b04a65a4
I20240501 14:09:35.953881 15925 ts_tablet_manager.cc:1909] T bfc871f734904165ab772321b04a65a4 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:35.954088 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 8a74c6b1b2dc4793a8d7ed4c41de013d (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.954403 15923 ts_tablet_manager.cc:1888] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:35.954468 15923 log.cc:1196] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/6305f3f7cb8846bf843d2d753fd1ae07
I20240501 14:09:35.955264 16520 raft_consensus.cc:1050] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8: attempting to promote NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 to VOTER
I20240501 14:09:35.955752 16520 consensus_queue.cc:237] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } }
I20240501 14:09:35.956427 15923 ts_tablet_manager.cc:1909] T 6305f3f7cb8846bf843d2d753fd1ae07 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:35.957077 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 6305f3f7cb8846bf843d2d753fd1ae07 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.957363 15353 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:35.957866 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet bfc871f734904165ab772321b04a65a4 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.957999 15088 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:35.958390 16184 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:35.958724 16184 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:35.959071 12585 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:35.960304 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a74c6b1b2dc4793a8d7ed4c41de013d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:35.960515 16379 tablet_replica.cc:330] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:35.960582 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc871f734904165ab772321b04a65a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:35 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:35.960654 16379 raft_consensus.cc:2229] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.960904 16379 raft_consensus.cc:2258] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.961565 16379 ts_tablet_manager.cc:1875] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.962217 16106 ts_tablet_manager.cc:1888] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:35.962299 16106 log.cc:1196] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/6305f3f7cb8846bf843d2d753fd1ae07
I20240501 14:09:35.962579 16106 ts_tablet_manager.cc:1909] T 6305f3f7cb8846bf843d2d753fd1ae07 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:35.963563 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 6305f3f7cb8846bf843d2d753fd1ae07 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.963879 16379 ts_tablet_manager.cc:1888] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.963953 16379 log.cc:1196] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a74c6b1b2dc4793a8d7ed4c41de013d
I20240501 14:09:35.964218 16379 ts_tablet_manager.cc:1909] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:35.964979 16379 tablet_replica.cc:330] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:35.965152 16379 raft_consensus.cc:2229] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:35.965216 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet bfc871f734904165ab772321b04a65a4 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.965327 16379 raft_consensus.cc:2258] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.965438 15922 tablet_replica.cc:330] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.965565 15922 raft_consensus.cc:2229] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.965691 15922 raft_consensus.cc:2258] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.965880 16379 ts_tablet_manager.cc:1875] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.966040 16723 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:35.966224 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 8a74c6b1b2dc4793a8d7ed4c41de013d (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.966814 15922 ts_tablet_manager.cc:1875] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.970546 16379 ts_tablet_manager.cc:1888] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:35.970945 16379 log.cc:1196] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/bfc871f734904165ab772321b04a65a4
I20240501 14:09:35.970999 15922 ts_tablet_manager.cc:1888] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:35.971097 15922 log.cc:1196] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/8a74c6b1b2dc4793a8d7ed4c41de013d
I20240501 14:09:35.971405 15922 ts_tablet_manager.cc:1909] T 8a74c6b1b2dc4793a8d7ed4c41de013d P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:35.971510 16379 ts_tablet_manager.cc:1909] T bfc871f734904165ab772321b04a65a4 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:35.971693 16184 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.972010 15922 tablet_replica.cc:330] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:35.972173 15922 raft_consensus.cc:2229] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.972337 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet bfc871f734904165ab772321b04a65a4 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.972484 15353 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.972580 15922 raft_consensus.cc:2258] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.973083 15922 ts_tablet_manager.cc:1875] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:35.973929 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 8a74c6b1b2dc4793a8d7ed4c41de013d (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.974857 12439 catalog_manager.cc:5484] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.974663 15088 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.975142 15922 ts_tablet_manager.cc:1888] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:35.975231 15922 log.cc:1196] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/6305f3f7cb8846bf843d2d753fd1ae07
I20240501 14:09:35.975483 15922 ts_tablet_manager.cc:1909] T 6305f3f7cb8846bf843d2d753fd1ae07 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:35.976246 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 6305f3f7cb8846bf843d2d753fd1ae07 (table rebalance_test_table_12 [id=09276a2b35f54b4cb28f878f0e3996f0]) successfully deleted
I20240501 14:09:35.978801 12585 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.981529 15221 consensus_queue.cc:237] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } }
I20240501 14:09:35.982594 12585 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.982648 15088 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:35.983012 16724 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.983191 16184 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:35.984622 16723 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.985105 15088 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.985117 12585 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:35.986483 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41a1a3342cae4af480f1e65164493219 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:35.986536 16474 raft_consensus.cc:482] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:35.986634 16474 raft_consensus.cc:504] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.987115 16474 leader_election.cc:288] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:35.987468 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5f1e720ec242c094590df7a6987918" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:35.987618 14957 raft_consensus.cc:2454] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:35.987998 15287 leader_election.cc:302] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:35.988180 16474 raft_consensus.cc:2781] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:35.988178 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5f1e720ec242c094590df7a6987918" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:35.988227 16474 raft_consensus.cc:482] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:35.988262 16474 raft_consensus.cc:3037] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.988284 15221 raft_consensus.cc:2454] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:35.988703 12439 catalog_manager.cc:5484] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:35.989212 16474 raft_consensus.cc:504] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.989516 16474 leader_election.cc:288] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:35.989825 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5f1e720ec242c094590df7a6987918" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:35.989840 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5f1e720ec242c094590df7a6987918" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:35.989929 15221 raft_consensus.cc:3037] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.989951 14957 raft_consensus.cc:3037] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:35.990718 14957 raft_consensus.cc:2454] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:35.990861 15221 raft_consensus.cc:2454] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:35.991080 15287 leader_election.cc:302] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:35.991266 16474 raft_consensus.cc:2781] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:35.991343 16474 raft_consensus.cc:686] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:35.991493 16474 consensus_queue.cc:237] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:35.993793 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a1a3342cae4af480f1e65164493219 with delete_type TABLET_DATA_TOMBSTONED (TS f23f426391724d6688aafffe7ee017d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:35.994280 12436 catalog_manager.cc:5484] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:35.994966 16729 tablet_replica.cc:330] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:35.995105 16729 raft_consensus.cc:2229] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:35.995224 16729 raft_consensus.cc:2258] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:35.995726 16729 ts_tablet_manager.cc:1875] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:35.997884 16729 ts_tablet_manager.cc:1888] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:35.997960 16729 log.cc:1196] T 41a1a3342cae4af480f1e65164493219 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/41a1a3342cae4af480f1e65164493219
I20240501 14:09:35.998507 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 41a1a3342cae4af480f1e65164493219 (table rebalancer_extra_table_0 [id=16400bc7338142aaa26d00036afb8d35]) successfully deleted
I20240501 14:09:36.023097 15425 raft_consensus.cc:482] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.023205 15425 raft_consensus.cc:504] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.023591 15425 leader_election.cc:288] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:36.024051 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:36.024199 15353 raft_consensus.cc:2454] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:36.024518 15026 leader_election.cc:302] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:36.024678 15425 raft_consensus.cc:2781] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.024719 15425 raft_consensus.cc:482] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.024750 15425 raft_consensus.cc:3037] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.024987 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:36.025106 15221 raft_consensus.cc:2454] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:36.025715 15425 raft_consensus.cc:504] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.026026 15425 leader_election.cc:288] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:36.026324 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:36.026456 15221 raft_consensus.cc:3037] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.026506 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:36.026643 15353 raft_consensus.cc:3037] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.027261 15221 raft_consensus.cc:2454] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:36.027451 15353 raft_consensus.cc:2454] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:36.027545 15023 leader_election.cc:302] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:36.027714 15425 raft_consensus.cc:2781] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.027802 15425 raft_consensus.cc:686] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:36.027952 15425 consensus_queue.cc:237] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.029659 12436 catalog_manager.cc:5484] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.046660 15422 raft_consensus.cc:482] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.046798 15422 raft_consensus.cc:504] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:36.047230 15422 leader_election.cc:288] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:36.047585 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:36.047703 12585 raft_consensus.cc:2454] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:36.047675 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:36.048032 15025 leader_election.cc:302] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:36.048333 15422 raft_consensus.cc:2781] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.048391 15422 raft_consensus.cc:482] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.048436 15422 raft_consensus.cc:3037] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.049319 15422 raft_consensus.cc:504] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:36.049659 15422 leader_election.cc:288] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:36.047801 15221 raft_consensus.cc:2454] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:36.049979 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:36.050012 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:36.050088 15222 raft_consensus.cc:3037] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.050112 12585 raft_consensus.cc:3037] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.050907 15222 raft_consensus.cc:2454] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:36.051136 12585 raft_consensus.cc:2454] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:36.051294 15023 leader_election.cc:302] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:36.051461 15422 raft_consensus.cc:2781] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.051553 15422 raft_consensus.cc:686] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:36.051690 15422 consensus_queue.cc:237] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:36.053333 12436 catalog_manager.cc:5484] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.077607 12827 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:36.079708 14938 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.086184 15200 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:09:36.088855 12696 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 14:09:36.098169 15069 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:09:36.099103 12958 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:36.101936 12566 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:36.107982 15331 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:36.118052 12436 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:36.123298 14957 raft_consensus.cc:1261] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.123467 15222 raft_consensus.cc:1261] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.123754 16475 consensus_queue.cc:1035] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.125856 16475 consensus_queue.cc:1035] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.126622 12436 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]). Expected version 1 got 0
I20240501 14:09:36.128166 12436 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]). Expected version 1 got 0
I20240501 14:09:36.128845 16532 mvcc.cc:205] Tried to move back new op lower bound from 7022889271794409472 to 7022889271262638080. Current Snapshot: MvccSnapshot[applied={T|T < 7022889271794409472}]
I20240501 14:09:36.130129 16509 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.130460 16509 tablet.cc:1620] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:36.130839 12436 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]). Expected version 1 got 0
I20240501 14:09:36.131574 12442 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]). Expected version 1 got 0
I20240501 14:09:36.132475 16748 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.132826 16748 tablet.cc:1620] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:36.135141 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab]). Expected version 1 got 0
I20240501 14:09:36.137169 12436 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab] alter complete (version 1)
I20240501 14:09:36.137990 16749 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.138437 16749 tablet.cc:1620] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:36.163431 16750 ts_tablet_manager.cc:902] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:36.164219 16750 tablet_copy_client.cc:308] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.5:33177
I20240501 14:09:36.165115 14978 tablet_copy_service.cc:140] P 8a13c028ba8a4a659b74af6c265df95e: Received BeginTabletCopySession request for tablet df57dee35f924801a23bd9b5c547a55f from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:53227)
I20240501 14:09:36.165235 14978 tablet_copy_service.cc:161] P 8a13c028ba8a4a659b74af6c265df95e: Beginning new tablet copy session on tablet df57dee35f924801a23bd9b5c547a55f from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:53227: session id = 4902d2609e6d44d089590a42e70356b2-df57dee35f924801a23bd9b5c547a55f
I20240501 14:09:36.166152 14978 tablet_copy_source_session.cc:205] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.167492 16750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df57dee35f924801a23bd9b5c547a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.169720 16750 tablet_copy_client.cc:783] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.170009 16750 tablet_copy_client.cc:647] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.171679 16750 tablet_copy_client.cc:523] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.173593 16750 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:36.175364 15353 consensus_queue.cc:237] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:36.176946 15088 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.176949 12718 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:36.177204 15288 consensus_peers.cc:485] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfce6c5fb6344a878935b322b5d8e2d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.177366 15571 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.177508 16475 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.178187 16750 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.178463 16750 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:09:36.178685 16750 ts_tablet_manager.cc:1367] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:09:36.179186 16750 raft_consensus.cc:348] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:36.179296 16750 raft_consensus.cc:729] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:09:36.179410 16750 consensus_queue.cc:260] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:36.179704 16750 ts_tablet_manager.cc:1398] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:36.179806 16474 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.180238 15088 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.182322 14978 tablet_copy_service.cc:342] P 8a13c028ba8a4a659b74af6c265df95e: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-df57dee35f924801a23bd9b5c547a55f received from {username='slave'} at 127.6.82.3:53227
I20240501 14:09:36.182387 14978 tablet_copy_service.cc:433] P 8a13c028ba8a4a659b74af6c265df95e: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-df57dee35f924801a23bd9b5c547a55f on tablet df57dee35f924801a23bd9b5c547a55f with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:09:36.182765 12439 catalog_manager.cc:5484] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.183666 12718 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.194679 12848 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.196094 15353 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:36.196231 12978 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:36.196519 16501 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:36.196810 16360 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:36.198208 16501 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.199024 15353 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.199565 12978 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
W20240501 14:09:36.200703 12782 consensus_peers.cc:485] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d64234effb3446bba1ff82bdf37acaa7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.202518 12436 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 6 to 7, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.213236 14957 consensus_queue.cc:237] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
W20240501 14:09:36.214665 14894 consensus_peers.cc:485] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba052327b37e4c4592dbf45a080fc5e1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.214814 15222 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.215209 16523 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.215520 15351 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.215967 16523 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.216877 16522 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:36.217489 15222 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:36.219224 15351 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:36.220310 12442 catalog_manager.cc:5484] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:36.239817 12978 consensus_queue.cc:237] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:36.240056 12442 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240501 14:09:36.241437 12914 consensus_peers.cc:485] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80afbd22fdad4bcdad3768c4a1953f0d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.242089 15351 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:36.242394 12718 raft_consensus.cc:1261] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:36.242520 16391 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:36.243101 16391 consensus_queue.cc:1035] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:36.244446 16375 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.244879 15351 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.248317 16748 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.248649 16748 tablet.cc:1620] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:36.249070 12718 raft_consensus.cc:2932] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.250169 16509 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.250481 16509 tablet.cc:1620] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:36.251452 12438 catalog_manager.cc:5484] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.250643 16749 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.252574 16749 tablet.cc:1620] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:36.257488 12436 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=891c7afd11f94d888914e4555c6266ab] alter complete (version 2)
I20240501 14:09:36.264863 12848 consensus_queue.cc:237] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.266498 12718 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.266650 12586 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.266947 16360 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.267308 16360 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.269011 16502 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
W20240501 14:09:36.270327 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 218f8a4fae324dc7bc535e6a18addb61. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.270978 12586 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.271060 12718 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.271993 12442 catalog_manager.cc:5484] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.277642 16793 ts_tablet_manager.cc:902] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:36.278256 16793 tablet_copy_client.cc:308] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:36.278971 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet a9f2c6558161436683f50486e437dc28 from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:58949)
I20240501 14:09:36.279071 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet a9f2c6558161436683f50486e437dc28 from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:58949: session id = f23f426391724d6688aafffe7ee017d8-a9f2c6558161436683f50486e437dc28
I20240501 14:09:36.279799 12869 tablet_copy_source_session.cc:205] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.280651 16793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f2c6558161436683f50486e437dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.282608 16793 tablet_copy_client.cc:783] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.282904 16793 tablet_copy_client.cc:647] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.283325 12586 consensus_queue.cc:237] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:36.284709 12848 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.284756 12978 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:36.284847 12523 consensus_peers.cc:485] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c3018b4356d462a9f2dd5511392661c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.285172 15655 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.285365 16290 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.285879 16793 tablet_copy_client.cc:523] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.287257 16793 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:36.287156 16366 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:36.287539 12848 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:36.289149 12978 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:36.290331 12436 catalog_manager.cc:5484] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:36.300812 12978 consensus_queue.cc:237] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.302425 12718 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:36.302696 12913 consensus_peers.cc:485] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56197e787ff14e85a07060f23e612498. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.302757 12848 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.302944 16375 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.303293 16375 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.305099 16391 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.305769 12718 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.307971 12848 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.308619 12442 catalog_manager.cc:5484] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.317083 16793 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.317399 16793 tablet_bootstrap.cc:492] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:36.317585 16793 ts_tablet_manager.cc:1367] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.000s
I20240501 14:09:36.317955 16793 raft_consensus.cc:348] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:36.318056 16793 raft_consensus.cc:729] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:36.318163 16793 consensus_queue.cc:260] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:36.318460 16793 ts_tablet_manager.cc:1398] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.319089 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-a9f2c6558161436683f50486e437dc28 received from {username='slave'} at 127.6.82.8:58949
I20240501 14:09:36.319077 15088 consensus_queue.cc:237] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:36.319175 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session f23f426391724d6688aafffe7ee017d8-a9f2c6558161436683f50486e437dc28 on tablet a9f2c6558161436683f50486e437dc28 with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:36.320590 12586 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.320883 14957 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:36.320912 15023 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c -> Peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): Couldn't send request to peer e6ab42bc38694f7f9fc8bfcc0a1483b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4010a078895473980e7108269ea7bc1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.321022 15422 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.321544 15422 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.322809 15425 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:36.323511 12586 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:36.323884 14957 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:36.326344 12436 catalog_manager.cc:5484] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:36.333012 15222 consensus_queue.cc:237] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:36.334331 14957 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.334360 15351 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:36.334650 15156 consensus_peers.cc:485] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37ab31238e6b4ea4b0da2561c79e7efd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.334823 16520 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.335063 16724 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.336091 16824 ts_tablet_manager.cc:902] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:36.336684 16824 tablet_copy_client.cc:308] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:36.336901 16723 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:36.337198 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58523)
I20240501 14:09:36.337302 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58523: session id = ab90e77f0027449bb3155980ca95593c-515d8f8e9d744aa4b2c8a4090c2f72e8
I20240501 14:09:36.337374 14957 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:36.337558 15351 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:36.338259 12738 tablet_copy_source_session.cc:205] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.339746 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.340150 12442 catalog_manager.cc:5484] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.344678 16824 tablet_copy_client.cc:783] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.344818 16824 tablet_copy_client.cc:647] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.346184 16824 tablet_copy_client.cc:523] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.347436 16824 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.349020 12718 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.350145 16750 ts_tablet_manager.cc:902] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:36.350477 12848 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.350656 16750 tablet_copy_client.cc:308] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:36.350848 12978 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.350889 15980 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.351271 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 188ce319682147ddaad30aae7232a311 from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:45837)
I20240501 14:09:36.351263 15980 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.351363 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 188ce319682147ddaad30aae7232a311 from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:45837: session id = 4902d2609e6d44d089590a42e70356b2-188ce319682147ddaad30aae7232a311
I20240501 14:09:36.352277 12999 tablet_copy_source_session.cc:205] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:09:36.352517 12653 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68b04992f1f94b36b01dfde24a42318d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.353016 15980 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.353467 16750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188ce319682147ddaad30aae7232a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.355522 12978 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.355801 16750 tablet_copy_client.cc:783] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.356132 12438 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.361129 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:36.365598 16750 tablet_copy_client.cc:647] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.369040 16750 tablet_copy_client.cc:523] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.370489 16750 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:36.374321 16748 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:36.379140 16749 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:36.383138 12848 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.384517 16509 tablet.cc:1722] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:36.385612 16824 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.385944 16824 tablet_bootstrap.cc:492] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:36.386250 16824 ts_tablet_manager.cc:1367] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.000s
I20240501 14:09:36.388906 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=891c7afd11f94d888914e4555c6266ab] alter complete (version 3)
I20240501 14:09:36.391120 16750 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.391479 16750 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:09:36.391376 16824 raft_consensus.cc:348] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.391547 16824 raft_consensus.cc:729] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:36.391733 16750 ts_tablet_manager.cc:1367] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240501 14:09:36.391733 16824 consensus_queue.cc:260] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.392237 16824 ts_tablet_manager.cc:1398] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:36.392282 16750 raft_consensus.cc:348] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:36.392421 16750 raft_consensus.cc:729] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:09:36.392534 16750 consensus_queue.cc:260] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:36.392973 16750 ts_tablet_manager.cc:1398] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.393741 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-188ce319682147ddaad30aae7232a311 received from {username='slave'} at 127.6.82.3:45837
I20240501 14:09:36.393822 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-188ce319682147ddaad30aae7232a311 on tablet 188ce319682147ddaad30aae7232a311 with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:09:36.393918 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-515d8f8e9d744aa4b2c8a4090c2f72e8 received from {username='slave'} at 127.6.82.6:58523
I20240501 14:09:36.393986 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session ab90e77f0027449bb3155980ca95593c-515d8f8e9d744aa4b2c8a4090c2f72e8 on tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:36.402664 14957 consensus_queue.cc:237] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:36.404469 15222 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:36.404798 14893 consensus_peers.cc:485] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dea946f39526401eb9b684f60cd9e8d0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.405144 16522 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.405228 15351 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.405596 16522 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:36.407310 16522 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.407826 15351 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.410010 15222 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.410523 12438 catalog_manager.cc:5484] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:09:36.421578 12848 consensus_queue.cc:237] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.423019 12586 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.423512 12718 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:36.423517 12782 consensus_peers.cc:485] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d01550698e8449b081a339f6ba30591c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.423655 16360 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.424165 16360 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.426111 16360 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.427091 12586 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.428951 12439 catalog_manager.cc:5484] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.436561 12718 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:36.461480 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:09:36.461634 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 14:09:36.461635 16305 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.464366 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 82a0a004a4ed4e05b4a099b1002372c6
I20240501 14:09:36.464709 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet a9f2c6558161436683f50486e437dc28
I20240501 14:09:36.464915 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 82a0a004a4ed4e05b4a099b1002372c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:36.464998 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet d2631324f66d454697b895ecdd227276
I20240501 14:09:36.465193 15922 tablet_replica.cc:330] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:36.465364 15922 raft_consensus.cc:2229] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.465538 15922 raft_consensus.cc:2258] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.465759 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet d2631324f66d454697b895ecdd227276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:36.465775 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f2c6558161436683f50486e437dc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:36.465812 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 82a0a004a4ed4e05b4a099b1002372c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:36.465937 16729 tablet_replica.cc:330] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:36.466084 16729 raft_consensus.cc:2229] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:36.466087 15922 ts_tablet_manager.cc:1875] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.466202 16729 raft_consensus.cc:2258] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:36.466682 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet d2631324f66d454697b895ecdd227276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:36.466830 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 82a0a004a4ed4e05b4a099b1002372c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:36.467103 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f2c6558161436683f50486e437dc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:36.467581 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet d2631324f66d454697b895ecdd227276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:36.467656 16729 ts_tablet_manager.cc:1875] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.467973 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f2c6558161436683f50486e437dc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:36.468468 16884 tablet_replica.cc:330] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:36.468614 16884 raft_consensus.cc:2229] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.468753 16884 raft_consensus.cc:2258] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.468777 15922 ts_tablet_manager.cc:1888] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.468860 15922 log.cc:1196] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/82a0a004a4ed4e05b4a099b1002372c6
I20240501 14:09:36.469132 15922 ts_tablet_manager.cc:1909] T 82a0a004a4ed4e05b4a099b1002372c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:36.469334 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f2c6558161436683f50486e437dc28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:36.469455 16884 ts_tablet_manager.cc:1875] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.469997 16379 tablet_replica.cc:330] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:36.470170 16379 raft_consensus.cc:2229] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.470445 15922 tablet_replica.cc:330] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:36.470448 16379 raft_consensus.cc:2258] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.470505 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 82a0a004a4ed4e05b4a099b1002372c6 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.470559 15922 raft_consensus.cc:2229] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.470685 15922 raft_consensus.cc:2258] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.471282 15922 ts_tablet_manager.cc:1875] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.472229 16884 ts_tablet_manager.cc:1888] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:36.472317 16884 log.cc:1196] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/d2631324f66d454697b895ecdd227276
I20240501 14:09:36.472575 16884 ts_tablet_manager.cc:1909] T d2631324f66d454697b895ecdd227276 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:36.473112 15922 ts_tablet_manager.cc:1888] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.473170 15922 log.cc:1196] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9f2c6558161436683f50486e437dc28
I20240501 14:09:36.473445 15922 ts_tablet_manager.cc:1909] T a9f2c6558161436683f50486e437dc28 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:36.473495 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet d2631324f66d454697b895ecdd227276 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.474022 16882 tablet_replica.cc:330] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:36.474165 16882 raft_consensus.cc:2229] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.474300 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet a9f2c6558161436683f50486e437dc28 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.474334 16882 raft_consensus.cc:2258] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.475167 16882 ts_tablet_manager.cc:1875] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.475420 16379 ts_tablet_manager.cc:1875] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.477402 16882 ts_tablet_manager.cc:1888] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:36.477485 16882 log.cc:1196] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/d2631324f66d454697b895ecdd227276
I20240501 14:09:36.477573 16379 ts_tablet_manager.cc:1888] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.477645 16379 log.cc:1196] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/a9f2c6558161436683f50486e437dc28
I20240501 14:09:36.477789 16882 ts_tablet_manager.cc:1909] T d2631324f66d454697b895ecdd227276 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:36.477938 16379 ts_tablet_manager.cc:1909] T a9f2c6558161436683f50486e437dc28 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:36.478366 16882 tablet_replica.cc:330] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:36.478616 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet d2631324f66d454697b895ecdd227276 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.478973 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet a9f2c6558161436683f50486e437dc28 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.479602 16729 ts_tablet_manager.cc:1888] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.479696 16729 log.cc:1196] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/a9f2c6558161436683f50486e437dc28
I20240501 14:09:36.480005 16729 ts_tablet_manager.cc:1909] T a9f2c6558161436683f50486e437dc28 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:36.480790 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet a9f2c6558161436683f50486e437dc28 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.481163 16883 tablet_replica.cc:330] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:36.481348 16883 raft_consensus.cc:2229] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.481657 16883 raft_consensus.cc:2258] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.482879 16882 raft_consensus.cc:2229] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.482929 16883 ts_tablet_manager.cc:1875] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.483071 16882 raft_consensus.cc:2258] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.483608 16882 ts_tablet_manager.cc:1875] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.485561 16883 ts_tablet_manager.cc:1888] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:36.485648 16883 log.cc:1196] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/d2631324f66d454697b895ecdd227276
I20240501 14:09:36.486238 16883 ts_tablet_manager.cc:1909] T d2631324f66d454697b895ecdd227276 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:36.487000 16883 tablet_replica.cc:330] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:36.487870 16883 raft_consensus.cc:2229] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.487897 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet d2631324f66d454697b895ecdd227276 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.488086 16883 raft_consensus.cc:2258] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.490327 16883 ts_tablet_manager.cc:1875] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.493109 16883 ts_tablet_manager.cc:1888] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.493202 16883 log.cc:1196] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/82a0a004a4ed4e05b4a099b1002372c6
I20240501 14:09:36.494163 16883 ts_tablet_manager.cc:1909] T 82a0a004a4ed4e05b4a099b1002372c6 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:36.487748 16837 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.495815 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:36.495956 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:36.498764 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 5e5f1e720ec242c094590df7a6987918
I20240501 14:09:36.499298 16882 ts_tablet_manager.cc:1888] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.499366 16882 log.cc:1196] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/82a0a004a4ed4e05b4a099b1002372c6
I20240501 14:09:36.499488 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5f1e720ec242c094590df7a6987918 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:36.499593 16882 ts_tablet_manager.cc:1909] T 82a0a004a4ed4e05b4a099b1002372c6 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:36.499562 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5f1e720ec242c094590df7a6987918 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:36.499881 16898 tablet_replica.cc:330] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:36.500128 16898 raft_consensus.cc:2229] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.500305 16898 raft_consensus.cc:2258] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.500591 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5f1e720ec242c094590df7a6987918 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:36.500789 16899 tablet_replica.cc:330] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:36.500823 16729 tablet_replica.cc:330] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:36.500890 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 82a0a004a4ed4e05b4a099b1002372c6 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.500974 16899 raft_consensus.cc:2229] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.500978 16729 raft_consensus.cc:2229] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.501096 16899 raft_consensus.cc:2258] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.501197 16729 raft_consensus.cc:2258] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.501308 16898 ts_tablet_manager.cc:1875] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.501669 16899 ts_tablet_manager.cc:1875] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.501983 16729 ts_tablet_manager.cc:1875] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.503811 16899 ts_tablet_manager.cc:1888] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.503880 16899 log.cc:1196] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/5e5f1e720ec242c094590df7a6987918
I20240501 14:09:36.504150 16899 ts_tablet_manager.cc:1909] T 5e5f1e720ec242c094590df7a6987918 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:36.504909 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table_new_name [id=891c7afd11f94d888914e4555c6266ab]) successfully deleted
I20240501 14:09:36.505311 16729 ts_tablet_manager.cc:1888] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.505473 16729 log.cc:1196] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/5e5f1e720ec242c094590df7a6987918
I20240501 14:09:36.506039 16729 ts_tablet_manager.cc:1909] T 5e5f1e720ec242c094590df7a6987918 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:36.506136 16898 ts_tablet_manager.cc:1888] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.506378 16898 log.cc:1196] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/5e5f1e720ec242c094590df7a6987918
I20240501 14:09:36.506886 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table_new_name [id=891c7afd11f94d888914e4555c6266ab]) successfully deleted
I20240501 14:09:36.507122 16898 ts_tablet_manager.cc:1909] T 5e5f1e720ec242c094590df7a6987918 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:36.507757 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 5e5f1e720ec242c094590df7a6987918 (table rebalancer_dynamic_table_new_name [id=891c7afd11f94d888914e4555c6266ab]) successfully deleted
I20240501 14:09:36.508298 16883 tablet_replica.cc:330] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:36.508416 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 82a0a004a4ed4e05b4a099b1002372c6 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.508610 16883 raft_consensus.cc:2229] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.508814 16883 raft_consensus.cc:2258] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.510087 16883 ts_tablet_manager.cc:1875] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.513013 16883 ts_tablet_manager.cc:1888] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:36.513161 16883 log.cc:1196] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9f2c6558161436683f50486e437dc28
I20240501 14:09:36.513499 16883 ts_tablet_manager.cc:1909] T a9f2c6558161436683f50486e437dc28 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:36.514822 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet a9f2c6558161436683f50486e437dc28 (table rebalance_test_table_14 [id=adf7f6bfbd534e6e9086a4008f9436e2]) successfully deleted
I20240501 14:09:36.539721 15425 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.544166 16732 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.549855 15422 consensus_queue.cc:1035] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.554929 16732 consensus_queue.cc:1035] T 5b037b6095a644fda854cf0365b1a7df P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.572641 12848 raft_consensus.cc:1203] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:36.601462 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:36.609791 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:36.610077 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab4add89cf1400080f812fd4caa8499 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]), partition=
I20240501 14:09:36.610376 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab4add89cf1400080f812fd4caa8499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.611609 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab4add89cf1400080f812fd4caa8499 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]), partition=
I20240501 14:09:36.611600 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab4add89cf1400080f812fd4caa8499 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]), partition=
I20240501 14:09:36.611824 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab4add89cf1400080f812fd4caa8499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.611822 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab4add89cf1400080f812fd4caa8499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.617712 16962 tablet_bootstrap.cc:492] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.618161 16961 tablet_bootstrap.cc:492] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:36.620066 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 565e6fb440dc4b00b3e2d0e1936889c8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.620100 15333 tablet_service.cc:1467] Processing CreateTablet for tablet 7130a0cb5bca433d8eb682e05b7a8f29 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.620334 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565e6fb440dc4b00b3e2d0e1936889c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.620410 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd8d72845814f17961145fd8d4d0bbd (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.619843 16962 tablet_bootstrap.cc:654] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.620601 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd8d72845814f17961145fd8d4d0bbd (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.620836 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8d72845814f17961145fd8d4d0bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.621737 16963 ts_tablet_manager.cc:902] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:36.622197 16963 tablet_copy_client.cc:308] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:36.622548 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 565e6fb440dc4b00b3e2d0e1936889c8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.622710 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 80afbd22fdad4bcdad3768c4a1953f0d from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59069)
I20240501 14:09:36.622799 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565e6fb440dc4b00b3e2d0e1936889c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.622830 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 80afbd22fdad4bcdad3768c4a1953f0d from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59069: session id = 8a13c028ba8a4a659b74af6c265df95e-80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:36.623761 12999 tablet_copy_source_session.cc:205] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.624949 16963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80afbd22fdad4bcdad3768c4a1953f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.626026 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 7130a0cb5bca433d8eb682e05b7a8f29 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.626129 15070 tablet_service.cc:1467] Processing CreateTablet for tablet 7130a0cb5bca433d8eb682e05b7a8f29 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.626272 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7130a0cb5bca433d8eb682e05b7a8f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.626359 15070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7130a0cb5bca433d8eb682e05b7a8f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.626818 16963 tablet_copy_client.cc:783] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.627065 16963 tablet_copy_client.cc:647] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.629096 16963 tablet_copy_client.cc:523] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.629103 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8d72845814f17961145fd8d4d0bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.630290 16963 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:36.626129 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd8d72845814f17961145fd8d4d0bbd (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.631042 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8d72845814f17961145fd8d4d0bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.631052 16972 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:36.631254 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 565e6fb440dc4b00b3e2d0e1936889c8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:36.631451 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565e6fb440dc4b00b3e2d0e1936889c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.631971 16972 tablet_bootstrap.cc:654] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.633831 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7130a0cb5bca433d8eb682e05b7a8f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.636626 16961 tablet_bootstrap.cc:654] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.637434 16971 ts_tablet_manager.cc:902] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Initiating tablet copy from peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:36.638214 16971 tablet_copy_client.cc:308] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.8:46801
I20240501 14:09:36.639050 16960 tablet_bootstrap.cc:492] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:36.639019 15373 tablet_copy_service.cc:140] P f23f426391724d6688aafffe7ee017d8: Received BeginTabletCopySession request for tablet bfce6c5fb6344a878935b322b5d8e2d0 from peer 6a3cc0660eed427eaee38164a8c1d4c3 ({username='slave'} at 127.6.82.1:52105)
I20240501 14:09:36.639149 15373 tablet_copy_service.cc:161] P f23f426391724d6688aafffe7ee017d8: Beginning new tablet copy session on tablet bfce6c5fb6344a878935b322b5d8e2d0 from peer 6a3cc0660eed427eaee38164a8c1d4c3 at {username='slave'} at 127.6.82.1:52105: session id = 6a3cc0660eed427eaee38164a8c1d4c3-bfce6c5fb6344a878935b322b5d8e2d0
I20240501 14:09:36.640287 16960 tablet_bootstrap.cc:654] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.640524 15373 tablet_copy_source_session.cc:205] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.643456 16961 tablet_bootstrap.cc:492] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:36.643620 16961 ts_tablet_manager.cc:1367] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:09:36.644170 16971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfce6c5fb6344a878935b322b5d8e2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.644227 16961 raft_consensus.cc:348] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.644356 16961 raft_consensus.cc:374] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.644425 16961 raft_consensus.cc:729] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.644573 16961 consensus_queue.cc:260] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.645438 16961 ts_tablet_manager.cc:1398] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.645675 16961 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:36.646665 16961 tablet_bootstrap.cc:654] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.646679 16970 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:36.648077 16970 tablet_bootstrap.cc:654] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.649281 16960 tablet_bootstrap.cc:492] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:36.649443 16960 ts_tablet_manager.cc:1367] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:36.650123 16960 raft_consensus.cc:348] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.650941 16960 raft_consensus.cc:374] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.651118 16960 raft_consensus.cc:729] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.651064 16971 tablet_copy_client.cc:783] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.651468 16971 tablet_copy_client.cc:647] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.651661 16523 raft_consensus.cc:482] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.651748 16523 raft_consensus.cc:504] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.652118 16523 leader_election.cc:288] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:36.651615 16960 consensus_queue.cc:260] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.652544 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab4add89cf1400080f812fd4caa8499" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:36.652572 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab4add89cf1400080f812fd4caa8499" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:36.652673 12586 raft_consensus.cc:2454] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
W20240501 14:09:36.653033 14893 leader_election.cc:341] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.655443 14893 leader_election.cc:302] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:36.655598 16523 raft_consensus.cc:2781] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.655659 16523 raft_consensus.cc:482] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.655700 16523 raft_consensus.cc:3037] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.655759 16971 tablet_copy_client.cc:523] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.656544 16960 ts_tablet_manager.cc:1398] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:36.656678 16523 raft_consensus.cc:504] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.657006 16523 leader_election.cc:288] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:36.657514 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab4add89cf1400080f812fd4caa8499" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:36.657519 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab4add89cf1400080f812fd4caa8499" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:36.657619 12586 raft_consensus.cc:3037] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:36.657938 14893 leader_election.cc:341] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.658705 12586 raft_consensus.cc:2454] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:36.659392 14893 leader_election.cc:302] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:36.659669 16523 raft_consensus.cc:2781] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.659756 16523 raft_consensus.cc:686] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:36.659899 16523 consensus_queue.cc:237] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.660785 16960 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:36.661216 12978 consensus_queue.cc:237] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:36.662163 12439 catalog_manager.cc:5484] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:36.663643 12848 raft_consensus.cc:1261] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:36.664721 12914 consensus_peers.cc:485] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cf9334f0f7444909b8e0ff366c71083. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.664861 16960 tablet_bootstrap.cc:654] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.665015 16391 consensus_queue.cc:1035] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.665736 16970 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:36.665896 16970 ts_tablet_manager.cc:1367] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:36.667395 16970 raft_consensus.cc:348] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.667783 16970 raft_consensus.cc:374] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.667881 16970 raft_consensus.cc:729] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.668095 16970 consensus_queue.cc:260] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.670013 16970 ts_tablet_manager.cc:1398] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:36.666647 12586 raft_consensus.cc:1261] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:36.670992 16971 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:36.671511 16391 consensus_queue.cc:1035] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:36.674285 16391 raft_consensus.cc:2932] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.677533 16963 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.677558 16972 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:36.677675 16972 ts_tablet_manager.cc:1367] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20240501 14:09:36.677853 16963 tablet_bootstrap.cc:492] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:36.678045 16963 ts_tablet_manager.cc:1367] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.048s user 0.021s sys 0.004s
I20240501 14:09:36.678267 16972 raft_consensus.cc:348] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.678407 16972 raft_consensus.cc:374] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.678457 16972 raft_consensus.cc:729] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.678493 16963 raft_consensus.cc:348] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:36.678640 16963 raft_consensus.cc:729] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:36.678627 16972 consensus_queue.cc:260] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.678759 16963 consensus_queue.cc:260] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:36.678761 12585 raft_consensus.cc:2932] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.679069 16963 ts_tablet_manager.cc:1398] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:36.680426 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-80afbd22fdad4bcdad3768c4a1953f0d received from {username='slave'} at 127.6.82.5:59069
I20240501 14:09:36.680502 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-80afbd22fdad4bcdad3768c4a1953f0d on tablet 80afbd22fdad4bcdad3768c4a1953f0d with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:36.681308 16972 ts_tablet_manager.cc:1398] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:36.681556 16972 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:36.682770 16972 tablet_bootstrap.cc:654] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.683800 12438 catalog_manager.cc:5484] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.686384 16972 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:36.686525 16972 ts_tablet_manager.cc:1367] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:09:36.687240 16972 raft_consensus.cc:348] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.687435 16972 raft_consensus.cc:374] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.687511 16972 raft_consensus.cc:729] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.687713 16972 consensus_queue.cc:260] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.688923 16750 ts_tablet_manager.cc:902] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:36.689488 16750 tablet_copy_client.cc:308] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.5:33177
I20240501 14:09:36.690225 14978 tablet_copy_service.cc:140] P 8a13c028ba8a4a659b74af6c265df95e: Received BeginTabletCopySession request for tablet ba052327b37e4c4592dbf45a080fc5e1 from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:53227)
I20240501 14:09:36.690354 14978 tablet_copy_service.cc:161] P 8a13c028ba8a4a659b74af6c265df95e: Beginning new tablet copy session on tablet ba052327b37e4c4592dbf45a080fc5e1 from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:53227: session id = 4902d2609e6d44d089590a42e70356b2-ba052327b37e4c4592dbf45a080fc5e1
I20240501 14:09:36.691272 16972 ts_tablet_manager.cc:1398] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:36.691519 14978 tablet_copy_source_session.cc:205] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.692219 16972 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:36.692776 16750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba052327b37e4c4592dbf45a080fc5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.693989 16972 tablet_bootstrap.cc:654] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.694641 16750 tablet_copy_client.cc:783] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.695506 16962 tablet_bootstrap.cc:492] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:36.695617 16962 ts_tablet_manager.cc:1367] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20240501 14:09:36.696017 16972 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:36.696089 16972 ts_tablet_manager.cc:1367] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:09:36.696208 16962 raft_consensus.cc:348] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.696336 16962 raft_consensus.cc:374] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.696384 16962 raft_consensus.cc:729] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.696494 16972 raft_consensus.cc:348] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.696525 16962 consensus_queue.cc:260] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.696609 16972 raft_consensus.cc:374] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.696645 16972 raft_consensus.cc:729] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.696784 16972 consensus_queue.cc:260] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.696878 16962 ts_tablet_manager.cc:1398] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.697057 16962 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.697062 16972 ts_tablet_manager.cc:1398] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.698233 16962 tablet_bootstrap.cc:654] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.698590 16750 tablet_copy_client.cc:647] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.701788 16960 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:36.701808 12848 raft_consensus.cc:2932] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:36.702072 16960 ts_tablet_manager.cc:1367] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240501 14:09:36.702723 16960 raft_consensus.cc:348] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.704645 16960 raft_consensus.cc:374] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.704748 16960 raft_consensus.cc:729] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.704958 16960 consensus_queue.cc:260] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.705375 16750 tablet_copy_client.cc:523] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.709236 16750 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:36.711966 15571 raft_consensus.cc:482] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.712059 12848 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.712077 15571 raft_consensus.cc:504] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.712574 15571 leader_election.cc:288] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:36.713341 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8d72845814f17961145fd8d4d0bbd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:36.713505 15221 raft_consensus.cc:2454] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:36.713519 12978 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:36.713519 15351 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:36.714035 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8d72845814f17961145fd8d4d0bbd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:36.714285 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:36.714507 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd5489eab5b04751bc8b3e5fb9e8cf73. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:36.714756 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:36.715183 15286 leader_election.cc:302] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
W20240501 14:09:36.715740 15288 leader_election.cc:341] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.716068 15571 raft_consensus.cc:2781] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.716130 15571 raft_consensus.cc:482] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.716179 15571 raft_consensus.cc:3037] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.716423 16960 ts_tablet_manager.cc:1398] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 14:09:36.717350 15571 raft_consensus.cc:504] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.717731 16961 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:36.717775 15571 leader_election.cc:288] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:36.717830 16961 ts_tablet_manager.cc:1367] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.000s
I20240501 14:09:36.718284 16961 raft_consensus.cc:348] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.718394 16961 raft_consensus.cc:374] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.718439 16961 raft_consensus.cc:729] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.718587 16961 consensus_queue.cc:260] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.718768 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8d72845814f17961145fd8d4d0bbd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:36.718901 16961 ts_tablet_manager.cc:1398] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.719225 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8d72845814f17961145fd8d4d0bbd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:36.719333 15221 raft_consensus.cc:3037] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.719519 16500 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.720430 15221 raft_consensus.cc:2454] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:36.720685 12978 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.720794 15286 leader_election.cc:302] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
W20240501 14:09:36.721204 15288 leader_election.cc:341] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:36.722894 16962 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:36.722998 16962 ts_tablet_manager.cc:1367] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240501 14:09:36.723577 16962 raft_consensus.cc:348] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.723695 16962 raft_consensus.cc:374] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.723565 12439 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.723731 16962 raft_consensus.cc:729] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.725703 16962 consensus_queue.cc:260] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.726135 15571 raft_consensus.cc:2781] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.726234 15571 raft_consensus.cc:686] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:36.726392 15571 consensus_queue.cc:237] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.728763 12438 catalog_manager.cc:5484] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:36.729871 16732 raft_consensus.cc:482] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.729974 16732 raft_consensus.cc:504] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:36.730417 16732 leader_election.cc:288] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:36.730893 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab4add89cf1400080f812fd4caa8499" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:36.731232 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab4add89cf1400080f812fd4caa8499" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:36.731388 12585 raft_consensus.cc:2379] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab90e77f0027449bb3155980ca95593c in current term 1: Already voted for candidate 8a13c028ba8a4a659b74af6c265df95e in this term.
I20240501 14:09:36.731406 16962 ts_tablet_manager.cc:1398] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:36.731650 15025 leader_election.cc:302] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab90e77f0027449bb3155980ca95593c; no voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:36.731882 16732 raft_consensus.cc:3037] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.733162 16732 raft_consensus.cc:2726] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:36.733553 16962 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.736665 16962 tablet_bootstrap.cc:654] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.738059 16962 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:36.738209 16962 ts_tablet_manager.cc:1367] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:36.738878 12848 raft_consensus.cc:1203] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:36.738932 16962 raft_consensus.cc:348] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.740322 16962 raft_consensus.cc:374] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.740454 16962 raft_consensus.cc:729] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.740693 16962 consensus_queue.cc:260] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:36.741884 16962 ts_tablet_manager.cc:1398] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:36.742156 16962 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.746922 16962 tablet_bootstrap.cc:654] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:36.748157 16971 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.748129 16962 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:36.748304 16962 ts_tablet_manager.cc:1367] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:09:36.748888 16971 tablet_bootstrap.cc:492] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap complete.
I20240501 14:09:36.749094 16971 ts_tablet_manager.cc:1367] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.000s
I20240501 14:09:36.749091 16962 raft_consensus.cc:348] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.749258 16962 raft_consensus.cc:374] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:36.749372 16962 raft_consensus.cc:729] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:36.749537 16962 consensus_queue.cc:260] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.749851 16962 ts_tablet_manager.cc:1398] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.750545 16824 ts_tablet_manager.cc:902] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:36.751045 17017 ts_tablet_manager.cc:902] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:36.751143 16824 tablet_copy_client.cc:308] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:36.752060 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 68b04992f1f94b36b01dfde24a42318d from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58523)
I20240501 14:09:36.752182 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 68b04992f1f94b36b01dfde24a42318d from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58523: session id = ab90e77f0027449bb3155980ca95593c-68b04992f1f94b36b01dfde24a42318d
I20240501 14:09:36.753297 17017 tablet_copy_client.cc:308] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:36.753469 12738 tablet_copy_source_session.cc:205] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.753741 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet d64234effb3446bba1ff82bdf37acaa7 from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:45273)
I20240501 14:09:36.753837 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet d64234effb3446bba1ff82bdf37acaa7 from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:45273: session id = 66e2dc34330848c3b8c2dd15898d34c8-d64234effb3446bba1ff82bdf37acaa7
I20240501 14:09:36.755169 12869 tablet_copy_source_session.cc:205] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.755474 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b04992f1f94b36b01dfde24a42318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.755990 17017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64234effb3446bba1ff82bdf37acaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.756521 16750 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.756850 16750 tablet_bootstrap.cc:492] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:09:36.757017 16750 ts_tablet_manager.cc:1367] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.001s
I20240501 14:09:36.757615 16750 raft_consensus.cc:348] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:36.757762 16750 raft_consensus.cc:729] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:09:36.757869 16750 consensus_queue.cc:260] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:36.757915 16824 tablet_copy_client.cc:783] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.758071 16824 tablet_copy_client.cc:647] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.758129 16750 ts_tablet_manager.cc:1398] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:36.759415 14978 tablet_copy_service.cc:342] P 8a13c028ba8a4a659b74af6c265df95e: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-ba052327b37e4c4592dbf45a080fc5e1 received from {username='slave'} at 127.6.82.3:53227
I20240501 14:09:36.759501 14978 tablet_copy_service.cc:433] P 8a13c028ba8a4a659b74af6c265df95e: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-ba052327b37e4c4592dbf45a080fc5e1 on tablet ba052327b37e4c4592dbf45a080fc5e1 with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:09:36.759675 16824 tablet_copy_client.cc:523] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.759819 17017 tablet_copy_client.cc:783] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.760125 17017 tablet_copy_client.cc:647] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.761142 16824 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.762354 17017 tablet_copy_client.cc:523] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.763756 17017 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:36.764087 17024 ts_tablet_manager.cc:902] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: Initiating tablet copy from peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:36.764364 15351 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:36.764637 17024 tablet_copy_client.cc:308] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.7:34847
I20240501 14:09:36.764698 16971 raft_consensus.cc:348] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:36.764814 16971 raft_consensus.cc:729] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: LEARNER
I20240501 14:09:36.764896 16971 consensus_queue.cc:260] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:36.765072 15242 tablet_copy_service.cc:140] P 66e2dc34330848c3b8c2dd15898d34c8: Received BeginTabletCopySession request for tablet 37ab31238e6b4ea4b0da2561c79e7efd from peer a6e94562bee149c2ace25672f0f9bf50 ({username='slave'} at 127.6.82.4:53055)
I20240501 14:09:36.765154 15242 tablet_copy_service.cc:161] P 66e2dc34330848c3b8c2dd15898d34c8: Beginning new tablet copy session on tablet 37ab31238e6b4ea4b0da2561c79e7efd from peer a6e94562bee149c2ace25672f0f9bf50 at {username='slave'} at 127.6.82.4:53055: session id = a6e94562bee149c2ace25672f0f9bf50-37ab31238e6b4ea4b0da2561c79e7efd
I20240501 14:09:36.765162 16971 ts_tablet_manager.cc:1398] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240501 14:09:36.766042 15242 tablet_copy_source_session.cc:205] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.766600 15373 tablet_copy_service.cc:342] P f23f426391724d6688aafffe7ee017d8: Request end of tablet copy session 6a3cc0660eed427eaee38164a8c1d4c3-bfce6c5fb6344a878935b322b5d8e2d0 received from {username='slave'} at 127.6.82.1:52105
I20240501 14:09:36.766708 15373 tablet_copy_service.cc:433] P f23f426391724d6688aafffe7ee017d8: ending tablet copy session 6a3cc0660eed427eaee38164a8c1d4c3-bfce6c5fb6344a878935b322b5d8e2d0 on tablet bfce6c5fb6344a878935b322b5d8e2d0 with peer 6a3cc0660eed427eaee38164a8c1d4c3
I20240501 14:09:36.767621 17024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab31238e6b4ea4b0da2561c79e7efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.770510 15089 raft_consensus.cc:1203] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:36.772053 17024 tablet_copy_client.cc:783] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.772203 17024 tablet_copy_client.cc:647] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.772395 12436 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:36.776748 17038 ts_tablet_manager.cc:902] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:36.777756 17038 tablet_copy_client.cc:308] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:36.778421 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 56197e787ff14e85a07060f23e612498 from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:34873)
I20240501 14:09:36.778517 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 56197e787ff14e85a07060f23e612498 from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:34873: session id = 66e2dc34330848c3b8c2dd15898d34c8-56197e787ff14e85a07060f23e612498
I20240501 14:09:36.779392 17024 tablet_copy_client.cc:523] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.779567 12999 tablet_copy_source_session.cc:205] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.780647 17024 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:36.781332 12585 raft_consensus.cc:1261] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.781927 15089 raft_consensus.cc:1261] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:36.783756 16523 consensus_queue.cc:1035] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.784057 16523 consensus_queue.cc:1035] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:36.784189 17024 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.784426 17024 tablet_bootstrap.cc:492] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: Bootstrap complete.
I20240501 14:09:36.784592 17024 ts_tablet_manager.cc:1367] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:09:36.784971 17024 raft_consensus.cc:348] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:36.785084 17024 raft_consensus.cc:729] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: LEARNER
I20240501 14:09:36.785176 17024 consensus_queue.cc:260] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:36.785454 17024 ts_tablet_manager.cc:1398] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:36.785718 15571 raft_consensus.cc:482] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.785816 15571 raft_consensus.cc:504] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.786204 15571 leader_election.cc:288] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:36.786345 15242 tablet_copy_service.cc:342] P 66e2dc34330848c3b8c2dd15898d34c8: Request end of tablet copy session a6e94562bee149c2ace25672f0f9bf50-37ab31238e6b4ea4b0da2561c79e7efd received from {username='slave'} at 127.6.82.4:53055
I20240501 14:09:36.786434 15242 tablet_copy_service.cc:433] P 66e2dc34330848c3b8c2dd15898d34c8: ending tablet copy session a6e94562bee149c2ace25672f0f9bf50-37ab31238e6b4ea4b0da2561c79e7efd on tablet 37ab31238e6b4ea4b0da2561c79e7efd with peer a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:36.788178 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565e6fb440dc4b00b3e2d0e1936889c8" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:36.788295 14957 raft_consensus.cc:2454] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:36.788512 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565e6fb440dc4b00b3e2d0e1936889c8" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:36.788640 15089 raft_consensus.cc:2454] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:36.789122 15287 leader_election.cc:302] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:36.790164 16824 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.790374 15571 raft_consensus.cc:2781] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.790441 15571 raft_consensus.cc:482] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.790479 16824 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:36.790488 15571 raft_consensus.cc:3037] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.790613 16824 ts_tablet_manager.cc:1367] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.000s
I20240501 14:09:36.791033 16824 raft_consensus.cc:348] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.791128 17038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56197e787ff14e85a07060f23e612498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.791204 16824 raft_consensus.cc:729] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:36.791314 16824 consensus_queue.cc:260] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.791405 12439 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]). Expected version 1 got 0
I20240501 14:09:36.791694 16824 ts_tablet_manager.cc:1398] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.791939 12438 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]). Expected version 1 got 0
I20240501 14:09:36.792305 15571 raft_consensus.cc:504] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.792531 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-68b04992f1f94b36b01dfde24a42318d received from {username='slave'} at 127.6.82.6:58523
I20240501 14:09:36.792604 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session ab90e77f0027449bb3155980ca95593c-68b04992f1f94b36b01dfde24a42318d on tablet 68b04992f1f94b36b01dfde24a42318d with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:36.792703 15571 leader_election.cc:288] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:36.793294 17038 tablet_copy_client.cc:783] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.793453 17038 tablet_copy_client.cc:647] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.793991 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565e6fb440dc4b00b3e2d0e1936889c8" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:36.794097 15089 raft_consensus.cc:3037] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.794996 15089 raft_consensus.cc:2454] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:36.795269 15288 leader_election.cc:302] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:36.795241 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565e6fb440dc4b00b3e2d0e1936889c8" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:36.795367 14957 raft_consensus.cc:3037] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.795635 17038 tablet_copy_client.cc:523] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.796344 14957 raft_consensus.cc:2454] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:36.796928 17038 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:36.797433 15571 raft_consensus.cc:2781] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.797541 15571 raft_consensus.cc:686] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:36.797677 15571 consensus_queue.cc:237] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.799711 12436 catalog_manager.cc:5484] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.801800 16509 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.802076 16509 tablet.cc:1620] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:36.802587 16793 ts_tablet_manager.cc:902] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:36.803197 16793 tablet_copy_client.cc:308] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:36.803864 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 0c3018b4356d462a9f2dd5511392661c from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:34877)
I20240501 14:09:36.803963 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 0c3018b4356d462a9f2dd5511392661c from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:34877: session id = f23f426391724d6688aafffe7ee017d8-0c3018b4356d462a9f2dd5511392661c
I20240501 14:09:36.805037 12607 tablet_copy_source_session.cc:205] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.805064 12439 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]). Expected version 1 got 0
I20240501 14:09:36.806078 16793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3018b4356d462a9f2dd5511392661c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.806533 12442 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573]). Expected version 1 got 0
I20240501 14:09:36.804818 17057 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.807366 17057 tablet.cc:1620] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:36.808171 16793 tablet_copy_client.cc:783] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.808312 16793 tablet_copy_client.cc:647] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.809341 16793 tablet_copy_client.cc:523] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.809445 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573] alter complete (version 1)
I20240501 14:09:36.810659 16793 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:36.812633 17052 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:36.813163 17052 tablet.cc:1620] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:36.825721 16824 ts_tablet_manager.cc:902] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:36.826213 16824 tablet_copy_client.cc:308] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:36.834303 16793 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.834679 16793 tablet_bootstrap.cc:492] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:36.834975 16793 ts_tablet_manager.cc:1367] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20240501 14:09:36.835533 16793 raft_consensus.cc:348] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:36.835733 16793 raft_consensus.cc:729] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:36.835906 16793 consensus_queue.cc:260] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:36.836285 16793 ts_tablet_manager.cc:1398] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:36.836846 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-0c3018b4356d462a9f2dd5511392661c received from {username='slave'} at 127.6.82.8:34877
I20240501 14:09:36.836946 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session f23f426391724d6688aafffe7ee017d8-0c3018b4356d462a9f2dd5511392661c on tablet 0c3018b4356d462a9f2dd5511392661c with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:36.838994 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet 218f8a4fae324dc7bc535e6a18addb61 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:51695)
I20240501 14:09:36.839092 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet 218f8a4fae324dc7bc535e6a18addb61 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:51695: session id = ab90e77f0027449bb3155980ca95593c-218f8a4fae324dc7bc535e6a18addb61
I20240501 14:09:36.839800 12869 tablet_copy_source_session.cc:205] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.840693 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 218f8a4fae324dc7bc535e6a18addb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.842432 16824 tablet_copy_client.cc:783] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.842577 16824 tablet_copy_client.cc:647] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.843691 16824 tablet_copy_client.cc:523] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.844659 16824 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:36.858055 17017 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.859071 16971 ts_tablet_manager.cc:902] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Initiating tablet copy from peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:36.859529 16971 tablet_copy_client.cc:308] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.5:33177
I20240501 14:09:36.860013 14978 tablet_copy_service.cc:140] P 8a13c028ba8a4a659b74af6c265df95e: Received BeginTabletCopySession request for tablet dea946f39526401eb9b684f60cd9e8d0 from peer 6a3cc0660eed427eaee38164a8c1d4c3 ({username='slave'} at 127.6.82.1:60155)
I20240501 14:09:36.860131 14978 tablet_copy_service.cc:161] P 8a13c028ba8a4a659b74af6c265df95e: Beginning new tablet copy session on tablet dea946f39526401eb9b684f60cd9e8d0 from peer 6a3cc0660eed427eaee38164a8c1d4c3 at {username='slave'} at 127.6.82.1:60155: session id = 6a3cc0660eed427eaee38164a8c1d4c3-dea946f39526401eb9b684f60cd9e8d0
I20240501 14:09:36.860110 15571 raft_consensus.cc:482] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.860204 15571 raft_consensus.cc:504] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.860574 15571 leader_election.cc:288] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:36.861083 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:36.861208 14978 tablet_copy_source_session.cc:205] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.861202 12586 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:36.861505 15288 leader_election.cc:302] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:36.861862 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:36.862007 15088 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:36.862210 15571 raft_consensus.cc:2781] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:36.862263 15571 raft_consensus.cc:482] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:36.862310 15571 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.862391 16971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea946f39526401eb9b684f60cd9e8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.863301 15571 raft_consensus.cc:504] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.863652 15571 leader_election.cc:288] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:36.864187 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:36.864290 12586 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.864709 16971 tablet_copy_client.cc:783] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.864840 16971 tablet_copy_client.cc:647] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.865195 12586 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:36.865535 15288 leader_election.cc:302] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:36.865945 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:36.866048 15088 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:36.866961 15088 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:36.867056 17017 tablet_bootstrap.cc:492] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:36.867252 17017 ts_tablet_manager.cc:1367] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.104s user 0.022s sys 0.004s
I20240501 14:09:36.867554 15571 raft_consensus.cc:2781] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:36.867636 15571 raft_consensus.cc:686] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:36.867758 15571 consensus_queue.cc:237] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:36.867799 17017 raft_consensus.cc:348] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.867900 17017 raft_consensus.cc:729] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:36.867992 17017 consensus_queue.cc:260] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.869647 16824 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.869648 12442 catalog_manager.cc:5484] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:36.869884 16824 tablet_bootstrap.cc:492] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:36.870025 16824 ts_tablet_manager.cc:1367] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20240501 14:09:36.870831 17017 ts_tablet_manager.cc:1398] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:36.872016 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-d64234effb3446bba1ff82bdf37acaa7 received from {username='slave'} at 127.6.82.7:45273
I20240501 14:09:36.872119 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-d64234effb3446bba1ff82bdf37acaa7 on tablet d64234effb3446bba1ff82bdf37acaa7 with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:36.874115 16824 raft_consensus.cc:348] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.874311 16824 raft_consensus.cc:729] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:36.874449 16824 consensus_queue.cc:260] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:36.876468 16824 ts_tablet_manager.cc:1398] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:09:36.878202 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-218f8a4fae324dc7bc535e6a18addb61 received from {username='slave'} at 127.6.82.6:51695
I20240501 14:09:36.878288 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session ab90e77f0027449bb3155980ca95593c-218f8a4fae324dc7bc535e6a18addb61 on tablet 218f8a4fae324dc7bc535e6a18addb61 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:36.879195 17038 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.879627 17038 tablet_bootstrap.cc:492] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:36.879676 16971 tablet_copy_client.cc:523] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.879851 17038 ts_tablet_manager.cc:1367] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.083s user 0.029s sys 0.000s
I20240501 14:09:36.880298 17038 raft_consensus.cc:348] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.880416 17038 raft_consensus.cc:729] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:36.880514 17038 consensus_queue.cc:260] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:36.881208 16971 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:36.881577 17038 ts_tablet_manager.cc:1398] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:36.882122 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-56197e787ff14e85a07060f23e612498 received from {username='slave'} at 127.6.82.7:34873
I20240501 14:09:36.882201 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-56197e787ff14e85a07060f23e612498 on tablet 56197e787ff14e85a07060f23e612498 with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:36.886363 16971 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.886619 16971 tablet_bootstrap.cc:492] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap complete.
I20240501 14:09:36.886802 16971 ts_tablet_manager.cc:1367] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20240501 14:09:36.887264 16971 raft_consensus.cc:348] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:36.887387 16971 raft_consensus.cc:729] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: LEARNER
I20240501 14:09:36.887490 16971 consensus_queue.cc:260] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:09:36.888722 16971 ts_tablet_manager.cc:1398] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:36.889207 14978 tablet_copy_service.cc:342] P 8a13c028ba8a4a659b74af6c265df95e: Request end of tablet copy session 6a3cc0660eed427eaee38164a8c1d4c3-dea946f39526401eb9b684f60cd9e8d0 received from {username='slave'} at 127.6.82.1:60155
I20240501 14:09:36.889282 14978 tablet_copy_service.cc:433] P 8a13c028ba8a4a659b74af6c265df95e: ending tablet copy session 6a3cc0660eed427eaee38164a8c1d4c3-dea946f39526401eb9b684f60cd9e8d0 on tablet dea946f39526401eb9b684f60cd9e8d0 with peer 6a3cc0660eed427eaee38164a8c1d4c3
I20240501 14:09:36.895701 17089 ts_tablet_manager.cc:902] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Initiating tablet copy from peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:36.896448 17089 tablet_copy_client.cc:308] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.6:37673
I20240501 14:09:36.897097 15110 tablet_copy_service.cc:140] P ab90e77f0027449bb3155980ca95593c: Received BeginTabletCopySession request for tablet f4010a078895473980e7108269ea7bc1 from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 ({username='slave'} at 127.6.82.2:41317)
I20240501 14:09:36.897213 15110 tablet_copy_service.cc:161] P ab90e77f0027449bb3155980ca95593c: Beginning new tablet copy session on tablet f4010a078895473980e7108269ea7bc1 from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 at {username='slave'} at 127.6.82.2:41317: session id = e6ab42bc38694f7f9fc8bfcc0a1483b7-f4010a078895473980e7108269ea7bc1
I20240501 14:09:36.898438 15110 tablet_copy_source_session.cc:205] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.900236 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4010a078895473980e7108269ea7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:36.903066 17089 tablet_copy_client.cc:783] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:36.903262 17089 tablet_copy_client.cc:647] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:36.904850 17089 tablet_copy_client.cc:523] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:36.906790 17089 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:36.912142 17089 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:36.912380 17089 tablet_bootstrap.cc:492] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap complete.
I20240501 14:09:36.912554 17089 ts_tablet_manager.cc:1367] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:09:36.912943 17089 raft_consensus.cc:348] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:36.913058 17089 raft_consensus.cc:729] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: LEARNER
I20240501 14:09:36.913159 17089 consensus_queue.cc:260] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:36.913496 12442 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:36.914518 17089 ts_tablet_manager.cc:1398] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:36.915696 15110 tablet_copy_service.cc:342] P ab90e77f0027449bb3155980ca95593c: Request end of tablet copy session e6ab42bc38694f7f9fc8bfcc0a1483b7-f4010a078895473980e7108269ea7bc1 received from {username='slave'} at 127.6.82.2:41317
I20240501 14:09:36.915771 15110 tablet_copy_service.cc:433] P ab90e77f0027449bb3155980ca95593c: ending tablet copy session e6ab42bc38694f7f9fc8bfcc0a1483b7-f4010a078895473980e7108269ea7bc1 on tablet f4010a078895473980e7108269ea7bc1 with peer e6ab42bc38694f7f9fc8bfcc0a1483b7
I20240501 14:09:36.920228 16509 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.920547 16509 tablet.cc:1620] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:36.920828 17057 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.921119 17057 tablet.cc:1620] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:36.921182 17052 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:36.921468 17052 tablet.cc:1620] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:36.925750 12436 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=3c5af602dcea4b11aa7a3603297c0573] alter complete (version 2)
I20240501 14:09:36.966598 12436 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:09:36.966784 12436 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 14:09:36.968814 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet e135742313954e889da54e332f689dd7
I20240501 14:09:36.969161 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet 80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:36.969439 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet e135742313954e889da54e332f689dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:36.969532 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet ca4e8d319cc54e6290d6f9feb54c9794
I20240501 14:09:36.969573 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet e135742313954e889da54e332f689dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:36.969626 15922 tablet_replica.cc:330] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:36.969760 15922 raft_consensus.cc:2229] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.969764 15333 tablet_service.cc:1514] Processing DeleteTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:36.969928 15922 raft_consensus.cc:2258] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.969959 16729 tablet_replica.cc:330] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:36.970070 16882 tablet_replica.cc:330] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:36.970137 16729 raft_consensus.cc:2229] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.970203 16882 raft_consensus.cc:2229] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.970279 16729 raft_consensus.cc:2258] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.970374 16882 raft_consensus.cc:2258] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.970469 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:36.970499 15922 ts_tablet_manager.cc:1875] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.970654 16899 tablet_replica.cc:330] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:36.970906 16882 ts_tablet_manager.cc:1875] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.971248 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:36.971305 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4e8d319cc54e6290d6f9feb54c9794 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:36.971423 16883 tablet_replica.cc:330] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:36.971547 16883 raft_consensus.cc:2229] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.971557 16899 raft_consensus.cc:2229] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:36.971668 16899 raft_consensus.cc:2258] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:36.971768 16883 raft_consensus.cc:2258] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.972546 12829 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4e8d319cc54e6290d6f9feb54c9794 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:36.972728 16379 tablet_replica.cc:330] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:36.972875 16379 raft_consensus.cc:2229] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.972932 15922 ts_tablet_manager.cc:1888] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.973001 15922 log.cc:1196] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/e135742313954e889da54e332f689dd7
I20240501 14:09:36.973013 16882 ts_tablet_manager.cc:1888] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.973034 16379 raft_consensus.cc:2258] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.973088 16882 log.cc:1196] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/e135742313954e889da54e332f689dd7
I20240501 14:09:36.973304 15922 ts_tablet_manager.cc:1909] T e135742313954e889da54e332f689dd7 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:36.973337 16882 ts_tablet_manager.cc:1909] T e135742313954e889da54e332f689dd7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:36.973610 16379 ts_tablet_manager.cc:1875] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.973971 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet e135742313954e889da54e332f689dd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:36.974877 16883 ts_tablet_manager.cc:1875] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.974979 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet e135742313954e889da54e332f689dd7 (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.975210 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4e8d319cc54e6290d6f9feb54c9794 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:36.975358 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 80afbd22fdad4bcdad3768c4a1953f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:36 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:36.975373 15922 tablet_replica.cc:330] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:36.975498 15922 raft_consensus.cc:2229] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.975622 15922 raft_consensus.cc:2258] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.975754 16379 ts_tablet_manager.cc:1888] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.975813 16379 log.cc:1196] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca4e8d319cc54e6290d6f9feb54c9794
I20240501 14:09:36.976017 16729 ts_tablet_manager.cc:1875] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.976059 16379 ts_tablet_manager.cc:1909] T ca4e8d319cc54e6290d6f9feb54c9794 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:36.976131 15922 ts_tablet_manager.cc:1875] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.976585 16379 tablet_replica.cc:330] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:36.976693 16379 raft_consensus.cc:2229] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.976819 16379 raft_consensus.cc:2258] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.977006 16883 ts_tablet_manager.cc:1888] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 14:09:36.977073 16883 log.cc:1196] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:36.977300 16379 ts_tablet_manager.cc:1875] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.977352 16883 ts_tablet_manager.cc:1909] T 80afbd22fdad4bcdad3768c4a1953f0d P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:36.977952 16883 tablet_replica.cc:330] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:36.978024 15922 ts_tablet_manager.cc:1888] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.978087 15922 log.cc:1196] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/ca4e8d319cc54e6290d6f9feb54c9794
I20240501 14:09:36.978091 16883 raft_consensus.cc:2229] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:36.978156 16729 ts_tablet_manager.cc:1888] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 14:09:36.978214 16729 log.cc:1196] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:36.978257 16883 raft_consensus.cc:2258] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.978332 15922 ts_tablet_manager.cc:1909] T ca4e8d319cc54e6290d6f9feb54c9794 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:36.978472 16729 ts_tablet_manager.cc:1909] T 80afbd22fdad4bcdad3768c4a1953f0d P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:36.978983 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 80afbd22fdad4bcdad3768c4a1953f0d (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.979146 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet e135742313954e889da54e332f689dd7 (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.979285 16882 tablet_replica.cc:330] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:36.979619 16882 raft_consensus.cc:2229] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:36.979904 16882 raft_consensus.cc:2258] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:36.980145 16899 ts_tablet_manager.cc:1875] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.980511 16883 ts_tablet_manager.cc:1875] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.980505 16882 ts_tablet_manager.cc:1875] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:36.981814 16899 ts_tablet_manager.cc:1888] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 14:09:36.981863 16899 log.cc:1196] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:36.982050 16899 ts_tablet_manager.cc:1909] T 80afbd22fdad4bcdad3768c4a1953f0d P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:36.982223 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet ca4e8d319cc54e6290d6f9feb54c9794 (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.982344 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 80afbd22fdad4bcdad3768c4a1953f0d (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.982429 16883 ts_tablet_manager.cc:1888] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.982446 16882 ts_tablet_manager.cc:1888] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.7
I20240501 14:09:36.982491 16883 log.cc:1196] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/ca4e8d319cc54e6290d6f9feb54c9794
I20240501 14:09:36.982620 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet ca4e8d319cc54e6290d6f9feb54c9794 (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.982677 16882 log.cc:1196] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/80afbd22fdad4bcdad3768c4a1953f0d
I20240501 14:09:36.982779 16883 ts_tablet_manager.cc:1909] T ca4e8d319cc54e6290d6f9feb54c9794 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:36.983110 16882 ts_tablet_manager.cc:1909] T 80afbd22fdad4bcdad3768c4a1953f0d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:36.983498 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet ca4e8d319cc54e6290d6f9feb54c9794 (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.983593 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 80afbd22fdad4bcdad3768c4a1953f0d (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.983851 16379 ts_tablet_manager.cc:1888] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:36.983912 16379 log.cc:1196] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/e135742313954e889da54e332f689dd7
I20240501 14:09:36.984061 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 80afbd22fdad4bcdad3768c4a1953f0d (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.984163 16379 ts_tablet_manager.cc:1909] T e135742313954e889da54e332f689dd7 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:36.984889 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet e135742313954e889da54e332f689dd7 (table rebalance_test_table_16 [id=9e4f852cf29b4335a258fb3e74f46eac]) successfully deleted
I20240501 14:09:36.996798 17038 ts_tablet_manager.cc:902] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:36.997311 17038 tablet_copy_client.cc:308] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:36.997808 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet d01550698e8449b081a339f6ba30591c from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:45273)
I20240501 14:09:36.997910 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet d01550698e8449b081a339f6ba30591c from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:45273: session id = 66e2dc34330848c3b8c2dd15898d34c8-d01550698e8449b081a339f6ba30591c
I20240501 14:09:36.998687 12869 tablet_copy_source_session.cc:205] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:36.999552 17038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d01550698e8449b081a339f6ba30591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.001374 17038 tablet_copy_client.cc:783] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:37.001523 17038 tablet_copy_client.cc:647] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:37.002614 17038 tablet_copy_client.cc:523] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:37.003664 17038 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:37.021353 17038 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:37.021683 17038 tablet_bootstrap.cc:492] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:37.021855 17038 ts_tablet_manager.cc:1367] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.007s
I20240501 14:09:37.022264 17038 raft_consensus.cc:348] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:37.022393 17038 raft_consensus.cc:729] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:37.022497 17038 consensus_queue.cc:260] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:37.022814 17038 ts_tablet_manager.cc:1398] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.023491 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-d01550698e8449b081a339f6ba30591c received from {username='slave'} at 127.6.82.7:45273
I20240501 14:09:37.023587 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-d01550698e8449b081a339f6ba30591c on tablet d01550698e8449b081a339f6ba30591c with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:37.029265 12436 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:37.038991 16509 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.040720 17057 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.043236 15425 raft_consensus.cc:482] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.043330 15425 raft_consensus.cc:504] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.043686 15425 leader_election.cc:288] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:37.044027 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8d72845814f17961145fd8d4d0bbd" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:37.044186 15221 raft_consensus.cc:2379] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab90e77f0027449bb3155980ca95593c in current term 1: Already voted for candidate f23f426391724d6688aafffe7ee017d8 in this term.
I20240501 14:09:37.044878 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8d72845814f17961145fd8d4d0bbd" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:37.045033 12436 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=3c5af602dcea4b11aa7a3603297c0573] alter complete (version 3)
I20240501 14:09:37.046564 15026 leader_election.cc:302] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab90e77f0027449bb3155980ca95593c; no voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8
I20240501 14:09:37.046764 15425 raft_consensus.cc:3037] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.047979 15425 raft_consensus.cc:2726] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:37.048974 17052 tablet.cc:1722] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.091584 12848 raft_consensus.cc:1203] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:37.108312 16771 raft_consensus.cc:1050] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: attempting to promote NON_VOTER 4902d2609e6d44d089590a42e70356b2 to VOTER
I20240501 14:09:37.108705 16771 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:37.109963 12848 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.109990 15221 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.110056 15088 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.110378 16771 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.110544 15421 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.110662 16523 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.112601 16771 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.113173 15088 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.114603 12848 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.115180 15221 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.115796 12436 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.120185 14957 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:37.121171 12848 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.121332 15088 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.121665 15895 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.121853 16523 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.123407 15421 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.125032 12430 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df57dee35f924801a23bd9b5c547a55f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.125173 12848 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.125313 15088 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.127013 12439 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.129805 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet df57dee35f924801a23bd9b5c547a55f with delete_type TABLET_DATA_TOMBSTONED (TS 66e2dc34330848c3b8c2dd15898d34c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:37.130154 17159 tablet_replica.cc:330] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:37.130332 17159 raft_consensus.cc:2229] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.130486 17159 raft_consensus.cc:2258] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.131016 17159 ts_tablet_manager.cc:1875] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.132632 17159 ts_tablet_manager.cc:1888] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.132700 17159 log.cc:1196] T df57dee35f924801a23bd9b5c547a55f P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/df57dee35f924801a23bd9b5c547a55f
I20240501 14:09:37.133165 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet df57dee35f924801a23bd9b5c547a55f (table rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]) successfully deleted
I20240501 14:09:37.137272 16475 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.140997 16475 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.150946 12436 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:37.151082 12436 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:37.152995 12436 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 7ab4add89cf1400080f812fd4caa8499
I20240501 14:09:37.153640 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab4add89cf1400080f812fd4caa8499 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:37.153632 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab4add89cf1400080f812fd4caa8499 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:37.153764 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab4add89cf1400080f812fd4caa8499 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:37.153846 16899 tablet_replica.cc:330] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:37.153932 15922 tablet_replica.cc:330] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:37.154017 17165 tablet_replica.cc:330] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:37.154012 16899 raft_consensus.cc:2229] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.154067 15922 raft_consensus.cc:2229] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.154124 17165 raft_consensus.cc:2229] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.154160 15922 raft_consensus.cc:2258] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.154219 16899 raft_consensus.cc:2258] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.154237 17165 raft_consensus.cc:2258] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.154816 15922 ts_tablet_manager.cc:1875] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.154855 16899 ts_tablet_manager.cc:1875] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.156868 15922 ts_tablet_manager.cc:1888] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.156941 15922 log.cc:1196] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/7ab4add89cf1400080f812fd4caa8499
I20240501 14:09:37.156936 15221 raft_consensus.cc:1203] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:37.157063 16899 ts_tablet_manager.cc:1888] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.157131 16899 log.cc:1196] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/7ab4add89cf1400080f812fd4caa8499
I20240501 14:09:37.157235 15922 ts_tablet_manager.cc:1909] T 7ab4add89cf1400080f812fd4caa8499 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:37.157405 16899 ts_tablet_manager.cc:1909] T 7ab4add89cf1400080f812fd4caa8499 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:37.157819 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table_new_name [id=3c5af602dcea4b11aa7a3603297c0573]) successfully deleted
I20240501 14:09:37.158277 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table_new_name [id=3c5af602dcea4b11aa7a3603297c0573]) successfully deleted
I20240501 14:09:37.158601 17165 ts_tablet_manager.cc:1875] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.160319 17165 ts_tablet_manager.cc:1888] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.160385 17165 log.cc:1196] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/7ab4add89cf1400080f812fd4caa8499
I20240501 14:09:37.160625 17165 ts_tablet_manager.cc:1909] T 7ab4add89cf1400080f812fd4caa8499 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:37.161290 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 7ab4add89cf1400080f812fd4caa8499 (table rebalancer_dynamic_table_new_name [id=3c5af602dcea4b11aa7a3603297c0573]) successfully deleted
I20240501 14:09:37.163081 16389 raft_consensus.cc:1050] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 4902d2609e6d44d089590a42e70356b2 to VOTER
I20240501 14:09:37.163442 16389 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:37.164645 14957 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.164750 15088 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.164752 12848 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.165161 16375 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.165349 16389 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.165560 17167 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.167477 16389 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.167872 12848 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.168936 16837 raft_consensus.cc:1050] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:37.169368 16837 consensus_queue.cc:237] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.170259 14957 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.171196 12436 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.171372 12586 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.171494 15089 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.171896 16837 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.174065 16305 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.174243 12848 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.174947 15088 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.175063 16837 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.176067 12978 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:37.177258 12849 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.177464 15090 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.177997 16375 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.178273 16389 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.180295 16389 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.181414 12849 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.182531 15090 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.182711 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 188ce319682147ddaad30aae7232a311 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.183815 16305 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.185187 12439 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.185794 12849 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.187850 12442 catalog_manager.cc:5484] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.188993 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 188ce319682147ddaad30aae7232a311 with delete_type TABLET_DATA_TOMBSTONED (TS 8a13c028ba8a4a659b74af6c265df95e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:37.188884 15090 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.189280 16899 tablet_replica.cc:330] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:37.189456 16899 raft_consensus.cc:2229] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.189628 16899 raft_consensus.cc:2258] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.189796 12586 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.190894 16899 ts_tablet_manager.cc:1875] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.194869 16899 ts_tablet_manager.cc:1888] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.194958 16899 log.cc:1196] T 188ce319682147ddaad30aae7232a311 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/188ce319682147ddaad30aae7232a311
I20240501 14:09:37.195878 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 188ce319682147ddaad30aae7232a311 (table rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]) successfully deleted
I20240501 14:09:37.197830 16824 ts_tablet_manager.cc:902] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:37.198292 16824 tablet_copy_client.cc:308] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:37.198858 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:51695)
I20240501 14:09:37.198976 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet bd5489eab5b04751bc8b3e5fb9e8cf73 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:51695: session id = ab90e77f0027449bb3155980ca95593c-bd5489eab5b04751bc8b3e5fb9e8cf73
I20240501 14:09:37.199693 12869 tablet_copy_source_session.cc:205] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:37.200748 16824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5489eab5b04751bc8b3e5fb9e8cf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.202785 16824 tablet_copy_client.cc:783] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:37.203644 16824 tablet_copy_client.cc:647] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:37.204957 16824 tablet_copy_client.cc:523] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:37.206104 16824 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:37.225242 15090 raft_consensus.cc:1203] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:37.225453 16475 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.231202 16475 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.238386 16824 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:37.238687 16824 tablet_bootstrap.cc:492] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:37.238888 16824 ts_tablet_manager.cc:1367] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240501 14:09:37.239724 16824 raft_consensus.cc:348] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:37.239859 16824 raft_consensus.cc:729] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:37.239949 16824 consensus_queue.cc:260] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:37.240232 16824 ts_tablet_manager.cc:1398] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.240619 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-bd5489eab5b04751bc8b3e5fb9e8cf73 received from {username='slave'} at 127.6.82.6:51695
I20240501 14:09:37.240684 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session ab90e77f0027449bb3155980ca95593c-bd5489eab5b04751bc8b3e5fb9e8cf73 on tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:37.245162 12585 raft_consensus.cc:1203] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:37.254509 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:37.259152 17234 ts_tablet_manager.cc:902] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:37.260022 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 56229e395cb3460b8364380705af3d39 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]), partition=
I20240501 14:09:37.260046 15333 tablet_service.cc:1467] Processing CreateTablet for tablet 56229e395cb3460b8364380705af3d39 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]), partition=
I20240501 14:09:37.260082 17234 tablet_copy_client.cc:308] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:37.260274 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56229e395cb3460b8364380705af3d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.260267 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56229e395cb3460b8364380705af3d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.260499 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 1cf9334f0f7444909b8e0ff366c71083 from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59069)
I20240501 14:09:37.260591 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 1cf9334f0f7444909b8e0ff366c71083 from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59069: session id = 8a13c028ba8a4a659b74af6c265df95e-1cf9334f0f7444909b8e0ff366c71083
I20240501 14:09:37.261503 12999 tablet_copy_source_session.cc:205] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:37.262140 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 56229e395cb3460b8364380705af3d39 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]), partition=
I20240501 14:09:37.262259 17234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf9334f0f7444909b8e0ff366c71083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.262319 17240 tablet_bootstrap.cc:492] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:37.263397 17240 tablet_bootstrap.cc:654] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.263708 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56229e395cb3460b8364380705af3d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.264364 17234 tablet_copy_client.cc:783] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:37.264510 17234 tablet_copy_client.cc:647] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:37.261991 17239 tablet_bootstrap.cc:492] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:37.266008 17234 tablet_copy_client.cc:523] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:37.266254 17240 tablet_bootstrap.cc:492] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:37.266347 17240 ts_tablet_manager.cc:1367] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:37.266938 17242 tablet_bootstrap.cc:492] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:37.267169 17234 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:37.267627 17240 raft_consensus.cc:348] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.267751 17240 raft_consensus.cc:374] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.267987 17242 tablet_bootstrap.cc:654] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.267805 17240 raft_consensus.cc:729] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.268522 17240 consensus_queue.cc:260] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.268862 17240 ts_tablet_manager.cc:1398] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.271464 17242 tablet_bootstrap.cc:492] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:37.271580 17242 ts_tablet_manager.cc:1367] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:37.271721 17239 tablet_bootstrap.cc:654] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.272056 17242 raft_consensus.cc:348] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.272159 17242 raft_consensus.cc:374] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.272194 17242 raft_consensus.cc:729] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.272320 17242 consensus_queue.cc:260] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.272620 17242 ts_tablet_manager.cc:1398] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.279829 17239 tablet_bootstrap.cc:492] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:37.280050 17239 ts_tablet_manager.cc:1367] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:37.280897 17239 raft_consensus.cc:348] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.281018 17239 raft_consensus.cc:374] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.281049 17239 raft_consensus.cc:729] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.282842 17239 consensus_queue.cc:260] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.283262 17239 ts_tablet_manager.cc:1398] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:37.284493 12978 raft_consensus.cc:1203] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:37.296608 17234 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:37.296883 17234 tablet_bootstrap.cc:492] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:37.297089 17234 ts_tablet_manager.cc:1367] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.003s
I20240501 14:09:37.297500 17234 raft_consensus.cc:348] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:37.297597 17234 raft_consensus.cc:729] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:37.297672 17234 consensus_queue.cc:260] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:37.298050 17234 ts_tablet_manager.cc:1398] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:37.298655 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-1cf9334f0f7444909b8e0ff366c71083 received from {username='slave'} at 127.6.82.5:59069
I20240501 14:09:37.298754 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-1cf9334f0f7444909b8e0ff366c71083 on tablet 1cf9334f0f7444909b8e0ff366c71083 with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:37.305051 12585 raft_consensus.cc:1203] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:37.306144 15221 raft_consensus.cc:1203] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:37.315019 15351 raft_consensus.cc:1203] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:37.320106 12718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54590
I20240501 14:09:37.320282 12718 raft_consensus.cc:595] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:37.332607 17223 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.333202 12718 consensus_queue.cc:237] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
W20240501 14:09:37.335300 12651 consensus_peers.cc:485] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40b29fb993f34e99ad3747a2ec7d6c4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:37.335784 17223 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.335968 12848 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.336717 16837 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.337153 12585 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.337527 16837 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.339340 15980 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:37.339813 12848 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:37.342351 12438 catalog_manager.cc:5484] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.343570 12585 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:37.351799 12718 consensus_queue.cc:237] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:37.353302 12978 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.353305 12848 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:37.353341 12654 consensus_peers.cc:485] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f1c8ff4149c42aa9517af442c38060c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:37.353744 15980 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.353919 17177 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.355762 16837 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:37.356220 12978 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:37.357983 12848 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:37.359243 12438 catalog_manager.cc:5484] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:37.376214 12436 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:37.382256 15333 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0546a5821a4bbf94007866029caa95 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.382380 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0546a5821a4bbf94007866029caa95 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.382584 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0546a5821a4bbf94007866029caa95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.382681 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0546a5821a4bbf94007866029caa95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.383426 15202 tablet_service.cc:1467] Processing CreateTablet for tablet e8226993467944dcb24d1efcda07538a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.383692 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8226993467944dcb24d1efcda07538a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.384788 17239 tablet_bootstrap.cc:492] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:37.385107 15200 tablet_service.cc:1467] Processing CreateTablet for tablet c2639132e2d04ce480edc800214f25be (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.385145 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0546a5821a4bbf94007866029caa95 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.385315 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2639132e2d04ce480edc800214f25be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.385442 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0546a5821a4bbf94007866029caa95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.385874 17239 tablet_bootstrap.cc:654] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.387013 12959 tablet_service.cc:1467] Processing CreateTablet for tablet e8226993467944dcb24d1efcda07538a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.387022 12698 tablet_service.cc:1467] Processing CreateTablet for tablet e8226993467944dcb24d1efcda07538a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.387239 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8226993467944dcb24d1efcda07538a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.387239 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8226993467944dcb24d1efcda07538a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.387887 17242 tablet_bootstrap.cc:492] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:37.388772 17242 tablet_bootstrap.cc:654] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.389482 17240 tablet_bootstrap.cc:492] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:37.390208 17240 tablet_bootstrap.cc:654] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.391448 14938 tablet_service.cc:1467] Processing CreateTablet for tablet c2639132e2d04ce480edc800214f25be (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.391752 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2639132e2d04ce480edc800214f25be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.391782 12958 tablet_service.cc:1467] Processing CreateTablet for tablet c2639132e2d04ce480edc800214f25be (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:37.391840 17320 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:37.391999 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2639132e2d04ce480edc800214f25be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.392673 17320 tablet_bootstrap.cc:654] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.393348 17323 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:37.394127 17323 tablet_bootstrap.cc:654] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.395143 17320 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:37.395637 17323 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:37.395785 17320 ts_tablet_manager.cc:1367] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:37.396181 17320 raft_consensus.cc:348] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.396311 17320 raft_consensus.cc:374] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.396435 17323 ts_tablet_manager.cc:1367] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:37.396807 17323 raft_consensus.cc:348] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.396898 17323 raft_consensus.cc:374] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.396930 17323 raft_consensus.cc:729] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.397059 17323 consensus_queue.cc:260] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.397337 17323 ts_tablet_manager.cc:1398] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:37.397409 17320 raft_consensus.cc:729] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.397557 17320 consensus_queue.cc:260] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.397817 17320 ts_tablet_manager.cc:1398] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:37.398162 17242 tablet_bootstrap.cc:492] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:37.398264 17242 ts_tablet_manager.cc:1367] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:37.398864 17320 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:37.398815 17242 raft_consensus.cc:348] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.398927 17242 raft_consensus.cc:374] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.398962 17242 raft_consensus.cc:729] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.399083 17242 consensus_queue.cc:260] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.399358 17242 ts_tablet_manager.cc:1398] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.399530 17242 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:37.399787 17320 tablet_bootstrap.cc:654] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.400445 17242 tablet_bootstrap.cc:654] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.403528 17240 tablet_bootstrap.cc:492] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:37.403633 17240 ts_tablet_manager.cc:1367] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:37.404147 17240 raft_consensus.cc:348] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.404278 17240 raft_consensus.cc:374] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.404315 17240 raft_consensus.cc:729] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.404449 17240 consensus_queue.cc:260] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.404732 17240 ts_tablet_manager.cc:1398] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.405884 17242 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:37.405977 17242 ts_tablet_manager.cc:1367] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:37.406526 17239 tablet_bootstrap.cc:492] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:37.406498 17242 raft_consensus.cc:348] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.406605 17242 raft_consensus.cc:374] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.406630 17239 ts_tablet_manager.cc:1367] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:37.406641 17242 raft_consensus.cc:729] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.406838 17242 consensus_queue.cc:260] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.407138 17242 ts_tablet_manager.cc:1398] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:37.407142 17239 raft_consensus.cc:348] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.407238 17239 raft_consensus.cc:374] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.407279 17239 raft_consensus.cc:729] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.407397 17239 consensus_queue.cc:260] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.407677 17239 ts_tablet_manager.cc:1398] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:37.407852 17239 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:37.408783 17239 tablet_bootstrap.cc:654] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.417542 15089 raft_consensus.cc:1203] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:37.418498 17239 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:37.418629 17239 ts_tablet_manager.cc:1367] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:37.419149 17239 raft_consensus.cc:348] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.419332 17239 raft_consensus.cc:374] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.419381 17239 raft_consensus.cc:729] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.419534 17239 consensus_queue.cc:260] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.419870 17239 ts_tablet_manager.cc:1398] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:37.420090 17239 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:37.421620 17239 tablet_bootstrap.cc:654] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:37.423949 17239 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:37.424036 17239 ts_tablet_manager.cc:1367] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:37.424410 17239 raft_consensus.cc:348] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.424484 17239 raft_consensus.cc:374] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.424515 17239 raft_consensus.cc:729] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.424655 17239 consensus_queue.cc:260] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.424875 17239 ts_tablet_manager.cc:1398] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:37.431432 17320 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:37.431607 17320 ts_tablet_manager.cc:1367] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.002s
I20240501 14:09:37.432637 17320 raft_consensus.cc:348] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.432835 17320 raft_consensus.cc:374] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:37.432896 17320 raft_consensus.cc:729] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:37.433099 17320 consensus_queue.cc:260] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.434042 17320 ts_tablet_manager.cc:1398] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:37.465485 12718 raft_consensus.cc:1203] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:37.473346 12442 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:09:37.473506 12442 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 14:09:37.476162 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 1a99b39e354144e48eee75f9f49a09fc
I20240501 14:09:37.476492 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 4 replicas of tablet 1cf9334f0f7444909b8e0ff366c71083
I20240501 14:09:37.476748 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99b39e354144e48eee75f9f49a09fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:37.476857 12442 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 717fa0889b89434ca145e573711b0677
I20240501 14:09:37.476953 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99b39e354144e48eee75f9f49a09fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:37.477110 17159 tablet_replica.cc:330] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:37.477124 16882 tablet_replica.cc:330] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:37.477115 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:37.477231 17159 raft_consensus.cc:2229] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.477229 16882 raft_consensus.cc:2229] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.477321 16883 tablet_replica.cc:330] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:37.477351 17159 raft_consensus.cc:2258] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.477434 16882 raft_consensus.cc:2258] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.477478 16883 raft_consensus.cc:2229] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.477758 16883 raft_consensus.cc:2258] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.477892 17159 ts_tablet_manager.cc:1875] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.477885 16882 ts_tablet_manager.cc:1875] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.477887 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:37.478220 16379 tablet_replica.cc:330] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:37.478214 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a99b39e354144e48eee75f9f49a09fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:37.478477 15922 tablet_replica.cc:330] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:37.478446 16379 raft_consensus.cc:2229] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.478636 15922 raft_consensus.cc:2229] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.478677 16379 raft_consensus.cc:2258] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.478915 15922 raft_consensus.cc:2258] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.478924 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:37.479107 16899 tablet_replica.cc:330] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:37.479230 16899 raft_consensus.cc:2229] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:09:37.479386 16899 raft_consensus.cc:2258] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:09:37.479539 15922 ts_tablet_manager.cc:1875] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.480021 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 717fa0889b89434ca145e573711b0677 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:37.480121 16882 ts_tablet_manager.cc:1888] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:37.480121 17159 ts_tablet_manager.cc:1888] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:37.480185 16882 log.cc:1196] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a99b39e354144e48eee75f9f49a09fc
I20240501 14:09:37.480185 17159 log.cc:1196] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/1a99b39e354144e48eee75f9f49a09fc
I20240501 14:09:37.480455 16882 ts_tablet_manager.cc:1909] T 1a99b39e354144e48eee75f9f49a09fc P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:37.480459 17159 ts_tablet_manager.cc:1909] T 1a99b39e354144e48eee75f9f49a09fc P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:37.480815 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf9334f0f7444909b8e0ff366c71083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:37.480829 16899 ts_tablet_manager.cc:1875] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.481823 15922 ts_tablet_manager.cc:1888] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240501 14:09:37.482503 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 717fa0889b89434ca145e573711b0677 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:37.482601 15922 log.cc:1196] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a99b39e354144e48eee75f9f49a09fc
I20240501 14:09:37.482920 15922 ts_tablet_manager.cc:1909] T 1a99b39e354144e48eee75f9f49a09fc P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:37.482965 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 717fa0889b89434ca145e573711b0677 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:37.483136 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 1a99b39e354144e48eee75f9f49a09fc (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.483428 16882 tablet_replica.cc:330] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:37.483502 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 1a99b39e354144e48eee75f9f49a09fc (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.483558 16882 raft_consensus.cc:2229] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.483611 15922 tablet_replica.cc:330] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:37.483688 16882 raft_consensus.cc:2258] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.483774 15922 raft_consensus.cc:2229] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.483799 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 1a99b39e354144e48eee75f9f49a09fc (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.483945 15922 raft_consensus.cc:2258] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.484138 16882 ts_tablet_manager.cc:1875] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.484334 16883 ts_tablet_manager.cc:1875] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.485653 16882 ts_tablet_manager.cc:1888] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.485723 16882 log.cc:1196] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/717fa0889b89434ca145e573711b0677
I20240501 14:09:37.485993 16882 ts_tablet_manager.cc:1909] T 717fa0889b89434ca145e573711b0677 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:37.486014 16883 ts_tablet_manager.cc:1888] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.486081 16883 log.cc:1196] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/1cf9334f0f7444909b8e0ff366c71083
I20240501 14:09:37.486397 16883 ts_tablet_manager.cc:1909] T 1cf9334f0f7444909b8e0ff366c71083 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:37.486635 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 717fa0889b89434ca145e573711b0677 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.487068 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 1cf9334f0f7444909b8e0ff366c71083 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.487773 15922 ts_tablet_manager.cc:1875] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.487841 16379 ts_tablet_manager.cc:1875] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.488600 15895 raft_consensus.cc:482] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.488673 15895 raft_consensus.cc:504] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.489001 15895 leader_election.cc:288] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:37.489413 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56229e395cb3460b8364380705af3d39" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:37.489528 15351 raft_consensus.cc:2454] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:37.489785 14894 leader_election.cc:302] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:37.489954 15895 raft_consensus.cc:2781] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.489976 16379 ts_tablet_manager.cc:1888] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.489996 15895 raft_consensus.cc:482] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.490041 16379 log.cc:1196] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/1cf9334f0f7444909b8e0ff366c71083
I20240501 14:09:37.490052 15895 raft_consensus.cc:3037] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.490299 16379 ts_tablet_manager.cc:1909] T 1cf9334f0f7444909b8e0ff366c71083 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:37.490626 15922 ts_tablet_manager.cc:1888] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.490691 15922 log.cc:1196] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/1cf9334f0f7444909b8e0ff366c71083
I20240501 14:09:37.490825 15895 raft_consensus.cc:504] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.490983 15922 ts_tablet_manager.cc:1909] T 1cf9334f0f7444909b8e0ff366c71083 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:37.491164 15895 leader_election.cc:288] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:37.491634 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56229e395cb3460b8364380705af3d39" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:37.491677 15922 tablet_replica.cc:330] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:37.491746 15351 raft_consensus.cc:3037] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.491804 15922 raft_consensus.cc:2229] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.491772 16899 ts_tablet_manager.cc:1888] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.491956 16899 log.cc:1196] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/1cf9334f0f7444909b8e0ff366c71083
I20240501 14:09:37.492003 15922 raft_consensus.cc:2258] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.492136 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 1cf9334f0f7444909b8e0ff366c71083 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.492214 16899 ts_tablet_manager.cc:1909] T 1cf9334f0f7444909b8e0ff366c71083 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:37.492534 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 1cf9334f0f7444909b8e0ff366c71083 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.492728 15351 raft_consensus.cc:2454] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:37.490567 16375 raft_consensus.cc:482] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.491902 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56229e395cb3460b8364380705af3d39" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:37.492846 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 1cf9334f0f7444909b8e0ff366c71083 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.492864 16375 raft_consensus.cc:504] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.492910 12979 raft_consensus.cc:3037] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.493119 14894 leader_election.cc:302] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:37.493201 16375 leader_election.cc:288] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:37.493288 15895 raft_consensus.cc:2781] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.493359 15895 raft_consensus.cc:686] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:37.491299 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56229e395cb3460b8364380705af3d39" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:37.493472 15895 consensus_queue.cc:237] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:37.493824 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2639132e2d04ce480edc800214f25be" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:37.493849 12979 raft_consensus.cc:2454] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:37.486971 16883 tablet_replica.cc:330] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:37.493939 14957 raft_consensus.cc:2454] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:37.494076 16883 raft_consensus.cc:2229] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.494315 12914 leader_election.cc:302] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:37.494362 16883 raft_consensus.cc:2258] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.494902 16883 ts_tablet_manager.cc:1875] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.494884 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2639132e2d04ce480edc800214f25be" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:37.495008 16375 raft_consensus.cc:2781] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.495010 15221 raft_consensus.cc:2454] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:37.495050 16375 raft_consensus.cc:482] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.495096 16375 raft_consensus.cc:3037] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.495963 16375 raft_consensus.cc:504] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.496165 12442 catalog_manager.cc:5484] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.496330 16375 leader_election.cc:288] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:37.496618 16883 ts_tablet_manager.cc:1888] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.496688 16883 log.cc:1196] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/717fa0889b89434ca145e573711b0677
I20240501 14:09:37.496696 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2639132e2d04ce480edc800214f25be" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:37.496793 14957 raft_consensus.cc:3037] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.497527 16883 ts_tablet_manager.cc:1909] T 717fa0889b89434ca145e573711b0677 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:37.497521 14957 raft_consensus.cc:2454] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:37.497735 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2639132e2d04ce480edc800214f25be" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:37.497850 15221 raft_consensus.cc:3037] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.498219 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 717fa0889b89434ca145e573711b0677 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.498888 15221 raft_consensus.cc:2454] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:37.499150 12913 leader_election.cc:302] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:37.499320 16375 raft_consensus.cc:2781] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.499413 16375 raft_consensus.cc:686] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:37.499537 16375 consensus_queue.cc:237] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.492727 15922 ts_tablet_manager.cc:1875] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.502641 15922 ts_tablet_manager.cc:1888] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 14:09:37.502439 12442 catalog_manager.cc:5484] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.502820 15922 log.cc:1196] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/717fa0889b89434ca145e573711b0677
I20240501 14:09:37.503162 15922 ts_tablet_manager.cc:1909] T 717fa0889b89434ca145e573711b0677 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:37.503767 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 717fa0889b89434ca145e573711b0677 (table rebalance_test_table_18 [id=290a56a7069c41b89e933c291498890f]) successfully deleted
I20240501 14:09:37.508472 16391 raft_consensus.cc:482] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.508569 16391 raft_consensus.cc:504] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.508939 16391 leader_election.cc:288] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:37.509490 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0546a5821a4bbf94007866029caa95" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:37.509618 15351 raft_consensus.cc:2454] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:37.509907 12916 leader_election.cc:302] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:37.510200 16391 raft_consensus.cc:2781] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.510242 16391 raft_consensus.cc:482] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.510267 16391 raft_consensus.cc:3037] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.511092 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0546a5821a4bbf94007866029caa95" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:37.511212 14957 raft_consensus.cc:2454] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:37.511252 16391 raft_consensus.cc:504] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.511595 16391 leader_election.cc:288] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:37.511965 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0546a5821a4bbf94007866029caa95" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:37.512073 14957 raft_consensus.cc:3037] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.513077 14957 raft_consensus.cc:2454] T 2b0546a5821a4bbf94007866029caa95 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:37.513078 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0546a5821a4bbf94007866029caa95" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:37.513216 15351 raft_consensus.cc:3037] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.513593 12914 leader_election.cc:302] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:37.513758 16391 raft_consensus.cc:2781] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.513840 16391 raft_consensus.cc:686] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:37.513968 16391 consensus_queue.cc:237] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.514173 15351 raft_consensus.cc:2454] T 2b0546a5821a4bbf94007866029caa95 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:37.515336 17160 raft_consensus.cc:482] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:37.515453 17160 raft_consensus.cc:504] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.515563 15221 raft_consensus.cc:1203] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:37.515880 17160 leader_election.cc:288] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:37.516490 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8226993467944dcb24d1efcda07538a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:37.516615 12979 raft_consensus.cc:2454] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:37.516896 15156 leader_election.cc:302] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:37.516785 12436 catalog_manager.cc:5484] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.517055 17160 raft_consensus.cc:2781] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:37.517107 17160 raft_consensus.cc:482] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:37.517143 17160 raft_consensus.cc:3037] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.518148 17160 raft_consensus.cc:504] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.518488 17160 leader_election.cc:288] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:37.519002 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8226993467944dcb24d1efcda07538a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:37.519104 12979 raft_consensus.cc:3037] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.520025 12979 raft_consensus.cc:2454] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:37.520416 15156 leader_election.cc:302] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:37.520622 17160 raft_consensus.cc:2781] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:37.520732 17160 raft_consensus.cc:686] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:37.520870 17160 consensus_queue.cc:237] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:37.522518 12442 catalog_manager.cc:5484] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.525314 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8226993467944dcb24d1efcda07538a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:37.525314 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8226993467944dcb24d1efcda07538a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:37.525482 12716 raft_consensus.cc:3037] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:37.525810 12718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54662
I20240501 14:09:37.525904 12718 raft_consensus.cc:595] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:37.526427 12716 raft_consensus.cc:2454] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:37.603482 12442 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:37.607683 12979 raft_consensus.cc:1261] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.607724 15351 raft_consensus.cc:1261] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:37.608074 16771 consensus_queue.cc:1035] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.608229 15895 consensus_queue.cc:1035] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.610957 12436 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]). Expected version 1 got 0
I20240501 14:09:37.611303 12438 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]). Expected version 1 got 0
I20240501 14:09:37.612315 17403 mvcc.cc:205] Tried to move back new op lower bound from 7022889277877227520 to 7022889277414715392. Current Snapshot: MvccSnapshot[applied={T|T < 7022889277877227520}]
I20240501 14:09:37.612550 17406 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:37.612910 17406 tablet.cc:1620] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:37.613333 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]). Expected version 1 got 0
I20240501 14:09:37.613955 17408 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:37.614066 12436 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]). Expected version 1 got 0
I20240501 14:09:37.614377 17408 tablet.cc:1620] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:37.616703 12436 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb]). Expected version 1 got 0
I20240501 14:09:37.617816 17407 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:37.619091 17407 tablet.cc:1620] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:37.620465 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb] alter complete (version 1)
I20240501 14:09:37.625557 15421 raft_consensus.cc:1050] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e: attempting to promote NON_VOTER 4902d2609e6d44d089590a42e70356b2 to VOTER
I20240501 14:09:37.625917 15421 consensus_queue.cc:237] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:37.627101 15351 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.627157 15221 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.627450 16523 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.627593 12848 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.627610 15895 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.628662 16523 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.630662 16523 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.632397 15221 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.632982 12848 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.633296 12438 catalog_manager.cc:5484] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.637372 14957 consensus_queue.cc:237] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:37.638325 12848 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.638345 15221 raft_consensus.cc:1261] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.638639 15421 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.638772 16771 consensus_queue.cc:1035] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.640221 15895 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.640722 15221 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.642428 12848 raft_consensus.cc:2932] T ba052327b37e4c4592dbf45a080fc5e1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.642938 12430 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba052327b37e4c4592dbf45a080fc5e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.643575 12436 catalog_manager.cc:5484] T ba052327b37e4c4592dbf45a080fc5e1 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:37.645969 15333 tablet_service.cc:1514] Processing DeleteTablet for tablet ba052327b37e4c4592dbf45a080fc5e1 with delete_type TABLET_DATA_TOMBSTONED (TS f23f426391724d6688aafffe7ee017d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:37.646375 17413 tablet_replica.cc:330] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:37.646514 17413 raft_consensus.cc:2229] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.646628 17413 pending_rounds.cc:62] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Trying to abort 1 pending ops.
I20240501 14:09:37.646783 17413 pending_rounds.cc:68] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022889277954879488 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ba052327b37e4c4592dbf45a080fc5e1" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } } }
I20240501 14:09:37.646919 17413 raft_consensus.cc:2894] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:09:37.646986 17413 raft_consensus.cc:2866] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:37.647037 17413 raft_consensus.cc:2258] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.647531 17413 ts_tablet_manager.cc:1875] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.649291 17413 ts_tablet_manager.cc:1888] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.649350 17413 log.cc:1196] T ba052327b37e4c4592dbf45a080fc5e1 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba052327b37e4c4592dbf45a080fc5e1
I20240501 14:09:37.649880 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet ba052327b37e4c4592dbf45a080fc5e1 (table rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]) successfully deleted
I20240501 14:09:37.667027 15980 raft_consensus.cc:979] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: : Instructing follower 6a3cc0660eed427eaee38164a8c1d4c3 to start an election
I20240501 14:09:37.667146 15980 raft_consensus.cc:1067] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Signalling peer 6a3cc0660eed427eaee38164a8c1d4c3 to start an election
I20240501 14:09:37.667672 12585 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8"
dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
from {username='slave'} at 127.6.82.2:41265
I20240501 14:09:37.667814 12585 raft_consensus.cc:482] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:37.667865 12585 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:37.668660 12585 raft_consensus.cc:504] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.669122 12585 leader_election.cc:288] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:37.669415 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:37.669414 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:37.669543 12848 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:37.669543 12716 raft_consensus.cc:3032] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:37.669606 12716 raft_consensus.cc:729] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:37.669602 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515d8f8e9d744aa4b2c8a4090c2f72e8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:37.669734 15089 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:37.669782 12716 consensus_queue.cc:260] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.669996 12716 raft_consensus.cc:3037] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:37.670540 15089 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 2.
I20240501 14:09:37.670586 12848 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 2.
I20240501 14:09:37.670949 12522 leader_election.cc:302] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:37.671039 12716 raft_consensus.cc:2454] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 2.
I20240501 14:09:37.671140 15655 raft_consensus.cc:2781] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:37.671262 15655 raft_consensus.cc:686] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:37.671411 15655 consensus_queue.cc:237] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.673214 12442 catalog_manager.cc:5484] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 1 to 2, leader changed from e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 2 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.703843 16360 raft_consensus.cc:1050] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:37.704260 16360 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:37.705561 15221 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:37.705700 12979 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:37.705907 15351 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:37.706014 15980 raft_consensus.cc:1050] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:37.706132 16502 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:37.706300 17415 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:37.706435 15980 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.706533 16502 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:37.707876 12848 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.708166 15089 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.708324 12978 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.708606 16305 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.708788 17415 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.708921 17177 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.709115 15351 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.709730 17177 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.711081 12978 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.711675 15221 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.711933 12442 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 7 to 8, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.712996 16837 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.715770 12848 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.716022 12436 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.716145 12849 consensus_queue.cc:237] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:37.717264 15351 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:37.717892 15878 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:37.718730 12979 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.719067 15221 raft_consensus.cc:1261] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:37.719422 16502 consensus_queue.cc:1035] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:37.720124 17415 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.720844 15089 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.721822 15351 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.722172 12716 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.723518 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:37.723891 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d64234effb3446bba1ff82bdf37acaa7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.724393 15089 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.725173 12442 catalog_manager.cc:5484] T d64234effb3446bba1ff82bdf37acaa7 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.725924 15980 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.727317 15221 raft_consensus.cc:2932] T d64234effb3446bba1ff82bdf37acaa7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.727522 12979 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.728061 15980 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.729190 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet d64234effb3446bba1ff82bdf37acaa7 with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:37.729598 16883 tablet_replica.cc:330] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:37.729737 16883 raft_consensus.cc:2229] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.729957 16883 raft_consensus.cc:2258] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.731128 17406 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:37.731479 17406 tablet.cc:1620] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:37.732331 17408 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:37.732604 17408 tablet.cc:1620] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:37.733258 16883 ts_tablet_manager.cc:1875] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.735565 17160 raft_consensus.cc:1050] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8: attempting to promote NON_VOTER a6e94562bee149c2ace25672f0f9bf50 to VOTER
I20240501 14:09:37.735853 15089 raft_consensus.cc:1203] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:37.736003 17160 consensus_queue.cc:237] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:37.736811 16883 ts_tablet_manager.cc:1888] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:37.736994 16883 log.cc:1196] T d64234effb3446bba1ff82bdf37acaa7 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/d64234effb3446bba1ff82bdf37acaa7
I20240501 14:09:37.737926 12437 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=6aa48234c9b04ff99ff096688441f2fb] alter complete (version 2)
I20240501 14:09:37.738468 14957 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.739703 16305 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.739998 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet d64234effb3446bba1ff82bdf37acaa7 (table rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]) successfully deleted
I20240501 14:09:37.740481 17160 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.740473 15090 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.740990 15351 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.740996 12979 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.741869 12979 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.742576 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68b04992f1f94b36b01dfde24a42318d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.742885 17428 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.743963 17407 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:37.744262 17407 tablet.cc:1620] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:37.744350 17428 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.745932 17428 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.746026 12436 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.747748 15351 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.748365 14957 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.748832 12438 catalog_manager.cc:5484] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.749894 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b04992f1f94b36b01dfde24a42318d with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:37.750427 16379 tablet_replica.cc:330] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:37.750582 16379 raft_consensus.cc:2229] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.750769 16379 raft_consensus.cc:2258] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.751309 15571 raft_consensus.cc:1050] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8: attempting to promote NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 to VOTER
I20240501 14:09:37.751430 16379 ts_tablet_manager.cc:1875] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.751745 15571 consensus_queue.cc:237] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } }
I20240501 14:09:37.753612 16379 ts_tablet_manager.cc:1888] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:37.754050 16379 log.cc:1196] T 68b04992f1f94b36b01dfde24a42318d P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/68b04992f1f94b36b01dfde24a42318d
I20240501 14:09:37.755003 12585 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.755502 17223 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.756405 12716 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.756455 15090 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.757519 17434 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.757771 15571 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.758286 12978 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.759395 15221 consensus_queue.cc:237] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:37.760689 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 68b04992f1f94b36b01dfde24a42318d (table rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]) successfully deleted
I20240501 14:09:37.761039 15351 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.761054 12978 raft_consensus.cc:1261] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.761381 16724 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.761758 17160 consensus_queue.cc:1035] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.762020 16475 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.762674 12716 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.763007 17160 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.763743 12585 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.764379 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37ab31238e6b4ea4b0da2561c79e7efd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.765128 12978 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.765141 12438 catalog_manager.cc:5484] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.766641 15351 raft_consensus.cc:2932] T 37ab31238e6b4ea4b0da2561c79e7efd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.767966 12436 catalog_manager.cc:5484] T 37ab31238e6b4ea4b0da2561c79e7efd P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 3 to 4, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:37.769227 15090 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.770429 15353 consensus_queue.cc:237] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } }
I20240501 14:09:37.771804 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ab31238e6b4ea4b0da2561c79e7efd with delete_type TABLET_DATA_TOMBSTONED (TS 8a13c028ba8a4a659b74af6c265df95e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:37.772104 12716 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.772408 16899 tablet_replica.cc:330] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:37.772562 16899 raft_consensus.cc:2229] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.772709 16899 raft_consensus.cc:2258] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.773505 16899 ts_tablet_manager.cc:1875] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.774328 12585 raft_consensus.cc:1261] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.775288 17434 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.775442 16475 consensus_queue.cc:1035] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.775727 16899 ts_tablet_manager.cc:1888] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.775808 16899 log.cc:1196] T 37ab31238e6b4ea4b0da2561c79e7efd P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/37ab31238e6b4ea4b0da2561c79e7efd
I20240501 14:09:37.776346 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 37ab31238e6b4ea4b0da2561c79e7efd (table rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]) successfully deleted
I20240501 14:09:37.777392 17223 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.777797 12585 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.778112 17167 raft_consensus.cc:1050] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:37.778470 17167 consensus_queue.cc:237] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:37.779135 12716 raft_consensus.cc:2932] T bfce6c5fb6344a878935b322b5d8e2d0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.780053 12718 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.780009 12438 catalog_manager.cc:5484] T bfce6c5fb6344a878935b322b5d8e2d0 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.781186 16375 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.781701 12848 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.781754 15221 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.782100 16375 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.782256 16391 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.783545 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfce6c5fb6344a878935b322b5d8e2d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.785100 15069 tablet_service.cc:1514] Processing DeleteTablet for tablet bfce6c5fb6344a878935b322b5d8e2d0 with delete_type TABLET_DATA_TOMBSTONED (TS ab90e77f0027449bb3155980ca95593c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:37.785589 16391 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.786162 17452 tablet_replica.cc:330] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:37.786288 17452 raft_consensus.cc:2229] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.786412 17452 raft_consensus.cc:2258] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.787535 12716 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.787513 12848 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.787552 15221 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.789945 12437 catalog_manager.cc:5484] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.786993 17452 ts_tablet_manager.cc:1875] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.793752 12978 consensus_queue.cc:237] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:37.794548 17452 ts_tablet_manager.cc:1888] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.794620 17452 log.cc:1196] T bfce6c5fb6344a878935b322b5d8e2d0 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/bfce6c5fb6344a878935b322b5d8e2d0
I20240501 14:09:37.794883 15221 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.795755 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet bfce6c5fb6344a878935b322b5d8e2d0 (table rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]) successfully deleted
I20240501 14:09:37.795799 17453 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.795249 17454 ts_tablet_manager.cc:902] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:37.796536 17454 tablet_copy_client.cc:308] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:37.796891 12716 raft_consensus.cc:1261] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.796995 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 1f1c8ff4149c42aa9517af442c38060c from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:35333)
I20240501 14:09:37.797082 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 1f1c8ff4149c42aa9517af442c38060c from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:35333: session id = f23f426391724d6688aafffe7ee017d8-1f1c8ff4149c42aa9517af442c38060c
I20240501 14:09:37.797638 16375 consensus_queue.cc:1035] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.798024 12738 tablet_copy_source_session.cc:205] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:37.798192 16391 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.798774 15221 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.801156 12438 catalog_manager.cc:5484] T 56197e787ff14e85a07060f23e612498 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.801641 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1c8ff4149c42aa9517af442c38060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.802181 12716 raft_consensus.cc:2932] T 56197e787ff14e85a07060f23e612498 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.802577 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56197e787ff14e85a07060f23e612498 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.807278 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 56197e787ff14e85a07060f23e612498 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:37.807480 16379 tablet_replica.cc:330] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:37.807615 16379 raft_consensus.cc:2229] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.807742 16379 raft_consensus.cc:2258] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.808686 16379 ts_tablet_manager.cc:1875] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.809057 17454 tablet_copy_client.cc:783] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:37.809222 17454 tablet_copy_client.cc:647] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:37.810336 17454 tablet_copy_client.cc:523] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:37.811256 16379 ts_tablet_manager.cc:1888] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:37.811328 16379 log.cc:1196] T 56197e787ff14e85a07060f23e612498 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/56197e787ff14e85a07060f23e612498
I20240501 14:09:37.811566 17454 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:37.812827 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 56197e787ff14e85a07060f23e612498 (table rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]) successfully deleted
I20240501 14:09:37.812995 17415 raft_consensus.cc:1050] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:37.813338 17415 consensus_queue.cc:237] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.814947 12718 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.814978 12585 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.815415 15090 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.815563 16360 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.815757 16502 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.815884 15878 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.818686 15878 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.820107 15090 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.820178 12718 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.822113 12436 catalog_manager.cc:5484] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.826263 12848 consensus_queue.cc:237] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:37.827309 15090 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.827679 15878 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.828224 12718 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.828557 15878 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.829130 16502 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.829506 15090 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.831058 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 218f8a4fae324dc7bc535e6a18addb61 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.831836 12436 catalog_manager.cc:5484] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.832335 12718 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:37.834756 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 218f8a4fae324dc7bc535e6a18addb61 with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:37.834975 15922 tablet_replica.cc:330] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:37.835108 15922 raft_consensus.cc:2229] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.835202 15922 pending_rounds.cc:62] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Trying to abort 1 pending ops.
I20240501 14:09:37.835247 15922 pending_rounds.cc:68] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022889278722523136 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "218f8a4fae324dc7bc535e6a18addb61" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } } }
I20240501 14:09:37.835397 15922 raft_consensus.cc:2894] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:09:37.835456 15922 raft_consensus.cc:2866] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:37.835495 15922 raft_consensus.cc:2258] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.836187 15922 ts_tablet_manager.cc:1875] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.838418 15922 ts_tablet_manager.cc:1888] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:37.838477 15922 log.cc:1196] T 218f8a4fae324dc7bc535e6a18addb61 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/218f8a4fae324dc7bc535e6a18addb61
I20240501 14:09:37.840791 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 218f8a4fae324dc7bc535e6a18addb61 (table rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]) successfully deleted
I20240501 14:09:37.840847 12436 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:37.843843 15655 raft_consensus.cc:1050] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:37.844244 15655 consensus_queue.cc:237] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:37.845818 15351 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.846231 17272 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.846235 17406 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.846793 17407 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.847504 12978 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.847915 17272 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.851800 12848 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.852357 15655 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.852754 12436 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=6aa48234c9b04ff99ff096688441f2fb] alter complete (version 3)
I20240501 14:09:37.852895 17408 tablet.cc:1722] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:37.857844 17454 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:37.858145 17454 tablet_bootstrap.cc:492] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:37.858307 17454 ts_tablet_manager.cc:1367] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.000s
I20240501 14:09:37.858682 17454 raft_consensus.cc:348] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:37.858919 17454 raft_consensus.cc:729] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:37.859071 17454 consensus_queue.cc:260] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:37.859555 17454 ts_tablet_manager.cc:1398] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:09:37.860131 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-1f1c8ff4149c42aa9517af442c38060c received from {username='slave'} at 127.6.82.8:35333
I20240501 14:09:37.860219 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session f23f426391724d6688aafffe7ee017d8-1f1c8ff4149c42aa9517af442c38060c on tablet 1f1c8ff4149c42aa9517af442c38060c with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:37.860401 17481 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.860527 12978 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.863111 15353 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.863698 12438 catalog_manager.cc:5484] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.872247 12848 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.873806 12585 consensus_queue.cc:237] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:37.875599 12848 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.875615 15353 raft_consensus.cc:1261] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.876036 17272 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.876310 15655 consensus_queue.cc:1035] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.877629 17481 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.879087 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0c3018b4356d462a9f2dd5511392661c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.880348 12848 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.880574 12442 catalog_manager.cc:5484] T 0c3018b4356d462a9f2dd5511392661c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:37.881851 15353 raft_consensus.cc:2932] T 0c3018b4356d462a9f2dd5511392661c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:37.885371 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c3018b4356d462a9f2dd5511392661c with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:37.885600 16883 tablet_replica.cc:330] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:37.885726 16883 raft_consensus.cc:2229] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.885944 16883 raft_consensus.cc:2258] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.888029 16883 ts_tablet_manager.cc:1875] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.890362 16883 ts_tablet_manager.cc:1888] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:37.890492 16883 log.cc:1196] T 0c3018b4356d462a9f2dd5511392661c P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/0c3018b4356d462a9f2dd5511392661c
I20240501 14:09:37.891189 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 0c3018b4356d462a9f2dd5511392661c (table rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]) successfully deleted
I20240501 14:09:37.892556 16732 raft_consensus.cc:1050] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c: attempting to promote NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 to VOTER
I20240501 14:09:37.893031 16732 consensus_queue.cc:237] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:37.894604 12718 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.894724 12585 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.894980 14957 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.895278 15422 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.895941 15422 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.896111 15425 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.898154 15425 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.899575 12585 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.899591 14957 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.900215 12718 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.900969 12436 catalog_manager.cc:5484] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.904703 15090 consensus_queue.cc:237] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:37.906421 12718 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.906842 16732 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.906919 14957 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.907613 16732 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.908993 15422 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.909401 14957 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.910661 12718 raft_consensus.cc:2932] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:37.911696 12438 catalog_manager.cc:5484] T f4010a078895473980e7108269ea7bc1 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 3 to 4, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.911906 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4010a078895473980e7108269ea7bc1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.914332 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet f4010a078895473980e7108269ea7bc1 with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:37.915532 15922 tablet_replica.cc:330] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:37.915679 15922 raft_consensus.cc:2229] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.915825 15922 raft_consensus.cc:2258] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.916663 15922 ts_tablet_manager.cc:1875] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.918668 15922 ts_tablet_manager.cc:1888] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.918756 15922 log.cc:1196] T f4010a078895473980e7108269ea7bc1 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/f4010a078895473980e7108269ea7bc1
I20240501 14:09:37.919422 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet f4010a078895473980e7108269ea7bc1 (table rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]) successfully deleted
I20240501 14:09:37.919999 16523 raft_consensus.cc:1050] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e: attempting to promote NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 to VOTER
I20240501 14:09:37.920302 16523 consensus_queue.cc:237] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } }
I20240501 14:09:37.921552 15221 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.921569 15353 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.921947 12585 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEARNER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:37.922014 15895 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.922714 15895 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.922950 16771 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:37.924860 15895 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.925336 12585 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.927211 15221 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.928129 12437 catalog_manager.cc:5484] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.929697 15353 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.933399 14957 consensus_queue.cc:237] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } }
I20240501 14:09:37.934875 15221 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.935242 12585 raft_consensus.cc:1261] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:37.935259 16771 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.935689 16771 consensus_queue.cc:1035] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:37.936436 17453 consensus_queue.cc:1035] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.937461 16771 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.939154 15221 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.940862 12430 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dea946f39526401eb9b684f60cd9e8d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.941239 12585 raft_consensus.cc:2932] T dea946f39526401eb9b684f60cd9e8d0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } }
I20240501 14:09:37.941758 12437 catalog_manager.cc:5484] T dea946f39526401eb9b684f60cd9e8d0 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 3 to 4, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:37.943907 17453 consensus_queue.cc:1035] T 2b0546a5821a4bbf94007866029caa95 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:37.946694 15333 tablet_service.cc:1514] Processing DeleteTablet for tablet dea946f39526401eb9b684f60cd9e8d0 with delete_type TABLET_DATA_TOMBSTONED (TS f23f426391724d6688aafffe7ee017d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:37.946949 17413 tablet_replica.cc:330] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:37.947111 17413 raft_consensus.cc:2229] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.947239 17413 raft_consensus.cc:2258] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.947723 17413 ts_tablet_manager.cc:1875] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.948843 17535 ts_tablet_manager.cc:902] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:37.949314 17535 tablet_copy_client.cc:308] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:37.949790 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 40b29fb993f34e99ad3747a2ec7d6c4f from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:35249)
I20240501 14:09:37.949906 17413 ts_tablet_manager.cc:1888] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:37.949910 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 40b29fb993f34e99ad3747a2ec7d6c4f from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:35249: session id = 66e2dc34330848c3b8c2dd15898d34c8-40b29fb993f34e99ad3747a2ec7d6c4f
I20240501 14:09:37.949978 17413 log.cc:1196] T dea946f39526401eb9b684f60cd9e8d0 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/dea946f39526401eb9b684f60cd9e8d0
I20240501 14:09:37.950570 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet dea946f39526401eb9b684f60cd9e8d0 (table rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]) successfully deleted
I20240501 14:09:37.950971 12738 tablet_copy_source_session.cc:205] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:37.951161 17415 raft_consensus.cc:1050] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:37.951611 17415 consensus_queue.cc:237] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:37.952073 17535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b29fb993f34e99ad3747a2ec7d6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:37.953251 12585 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.953296 12718 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.953455 15221 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:37.953647 15878 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.954246 16502 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.954696 15878 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:37.955152 17535 tablet_copy_client.cc:783] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:37.955292 17535 tablet_copy_client.cc:647] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:37.955711 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:37.955842 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:37.957803 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 56229e395cb3460b8364380705af3d39
I20240501 14:09:37.958302 17535 tablet_copy_client.cc:523] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:37.958796 15333 tablet_service.cc:1514] Processing DeleteTablet for tablet 56229e395cb3460b8364380705af3d39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:37.958990 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 56229e395cb3460b8364380705af3d39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:37.959197 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 56229e395cb3460b8364380705af3d39 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:37 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:37.959388 17413 tablet_replica.cc:330] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:37.959527 17413 raft_consensus.cc:2229] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.959645 17413 raft_consensus.cc:2258] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.959578 15878 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.959808 17535 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:37.959869 16883 tablet_replica.cc:330] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:37.960032 16883 raft_consensus.cc:2229] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.960148 16883 raft_consensus.cc:2258] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.960304 16899 tablet_replica.cc:330] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:37.960420 16899 raft_consensus.cc:2229] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:37.960654 16899 raft_consensus.cc:2258] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.961565 12585 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.962416 17413 ts_tablet_manager.cc:1875] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.963915 15221 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.964267 12438 catalog_manager.cc:5484] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.964459 17413 ts_tablet_manager.cc:1888] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.964535 17413 log.cc:1196] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/56229e395cb3460b8364380705af3d39
I20240501 14:09:37.964838 17413 ts_tablet_manager.cc:1909] T 56229e395cb3460b8364380705af3d39 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:37.965978 16883 ts_tablet_manager.cc:1875] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.966362 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table_new_name [id=6aa48234c9b04ff99ff096688441f2fb]) successfully deleted
I20240501 14:09:37.967832 12848 consensus_queue.cc:237] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:37.968191 16883 ts_tablet_manager.cc:1888] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.968255 16883 log.cc:1196] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/56229e395cb3460b8364380705af3d39
I20240501 14:09:37.968494 16883 ts_tablet_manager.cc:1909] T 56229e395cb3460b8364380705af3d39 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:37.969151 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table_new_name [id=6aa48234c9b04ff99ff096688441f2fb]) successfully deleted
I20240501 14:09:37.969048 12718 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.969513 12585 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.969584 15221 raft_consensus.cc:1261] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:37.969848 16502 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.970345 16502 consensus_queue.cc:1035] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:37.971444 16502 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.972236 15221 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.973023 12585 raft_consensus.cc:2932] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.973461 16899 ts_tablet_manager.cc:1875] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:37.975656 16899 ts_tablet_manager.cc:1888] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:37.975721 16899 log.cc:1196] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/56229e395cb3460b8364380705af3d39
I20240501 14:09:37.975998 16899 ts_tablet_manager.cc:1909] T 56229e395cb3460b8364380705af3d39 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:37.976612 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 56229e395cb3460b8364380705af3d39 (table rebalancer_dynamic_table_new_name [id=6aa48234c9b04ff99ff096688441f2fb]) successfully deleted
I20240501 14:09:37.977135 12437 catalog_manager.cc:5484] T d01550698e8449b081a339f6ba30591c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:37.977871 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d01550698e8449b081a339f6ba30591c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:37.979527 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet d01550698e8449b081a339f6ba30591c with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:37.979751 16882 tablet_replica.cc:330] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:37.979909 16882 raft_consensus.cc:2229] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:37.980080 16882 raft_consensus.cc:2258] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:37.982436 16882 ts_tablet_manager.cc:1875] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:37.984364 16882 ts_tablet_manager.cc:1888] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:37.984433 16882 log.cc:1196] T d01550698e8449b081a339f6ba30591c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/d01550698e8449b081a339f6ba30591c
I20240501 14:09:37.984947 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet d01550698e8449b081a339f6ba30591c (table rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]) successfully deleted
I20240501 14:09:37.998679 17535 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:37.999022 17535 tablet_bootstrap.cc:492] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:37.999178 17535 ts_tablet_manager.cc:1367] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.012s
I20240501 14:09:37.999567 17535 raft_consensus.cc:348] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:37.999673 17535 raft_consensus.cc:729] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:37.999747 17535 consensus_queue.cc:260] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:37.999997 17535 ts_tablet_manager.cc:1398] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.000520 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-40b29fb993f34e99ad3747a2ec7d6c4f received from {username='slave'} at 127.6.82.7:35249
I20240501 14:09:38.000593 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-40b29fb993f34e99ad3747a2ec7d6c4f on tablet 40b29fb993f34e99ad3747a2ec7d6c4f with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:38.007249 12848 consensus_queue.cc:237] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.008772 12716 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.008968 12978 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.009119 15878 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.009380 16360 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:38.009562 12784 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5957da39df64253a2b8d7f388308719. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.010866 15878 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.011250 12716 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.011926 12978 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.013710 12438 catalog_manager.cc:5484] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.023324 12585 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.024899 14957 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:38.025087 12523 consensus_peers.cc:485] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d178c3291efc44369e3adb921ccc6517. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.025085 12978 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:38.025391 17481 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.025594 15655 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.027262 17480 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.027616 14957 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.027607 12978 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.030246 12442 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.039470 12585 consensus_queue.cc:237] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:38.040796 12848 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.041018 12716 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.041321 12520 consensus_peers.cc:485] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 444c1a9d0e6943dc880161ae320673b6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.041332 17481 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.041513 15655 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.043174 17272 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:38.043613 12716 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:38.044872 12437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:38.046905 12439 catalog_manager.cc:5484] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.046950 12848 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:38.053757 15069 tablet_service.cc:1467] Processing CreateTablet for tablet 79e8de8cd4d2407c99ea4c5520d007f5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.054100 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e8de8cd4d2407c99ea4c5520d007f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.054242 12958 tablet_service.cc:1467] Processing CreateTablet for tablet a447cf75eb324846858ca40d152ea573 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.054543 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a447cf75eb324846858ca40d152ea573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.054664 15068 tablet_service.cc:1467] Processing CreateTablet for tablet b49fa7fe41af439688ef801bb2044485 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.054915 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49fa7fe41af439688ef801bb2044485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.054879 14938 tablet_service.cc:1467] Processing CreateTablet for tablet a447cf75eb324846858ca40d152ea573 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.055120 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a447cf75eb324846858ca40d152ea573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.056757 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 79e8de8cd4d2407c99ea4c5520d007f5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.057008 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e8de8cd4d2407c99ea4c5520d007f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.057063 14937 tablet_service.cc:1467] Processing CreateTablet for tablet b49fa7fe41af439688ef801bb2044485 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.057314 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49fa7fe41af439688ef801bb2044485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.058367 15333 tablet_service.cc:1467] Processing CreateTablet for tablet b49fa7fe41af439688ef801bb2044485 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.058512 17582 tablet_bootstrap.cc:492] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.058643 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49fa7fe41af439688ef801bb2044485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.058905 17580 tablet_bootstrap.cc:492] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:38.059666 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 79e8de8cd4d2407c99ea4c5520d007f5 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.059957 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e8de8cd4d2407c99ea4c5520d007f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.060690 17580 tablet_bootstrap.cc:654] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.061589 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:38.062292 17453 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.063060 17582 tablet_bootstrap.cc:654] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.064013 17585 tablet_bootstrap.cc:492] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:38.064155 15200 tablet_service.cc:1467] Processing CreateTablet for tablet a447cf75eb324846858ca40d152ea573 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ddfaca790bbb4e68b4fdf0185c62aff7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.064424 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a447cf75eb324846858ca40d152ea573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.065095 17583 tablet_bootstrap.cc:492] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:38.066124 17583 tablet_bootstrap.cc:654] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.066942 17581 tablet_bootstrap.cc:492] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:38.068055 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 095cff740f14479a962d7c077bc53908 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]), partition=
I20240501 14:09:38.068248 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095cff740f14479a962d7c077bc53908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.068392 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 095cff740f14479a962d7c077bc53908 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]), partition=
I20240501 14:09:38.068393 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 095cff740f14479a962d7c077bc53908 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]), partition=
I20240501 14:09:38.068586 17585 tablet_bootstrap.cc:654] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.068610 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095cff740f14479a962d7c077bc53908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.068609 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095cff740f14479a962d7c077bc53908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.069622 17580 tablet_bootstrap.cc:492] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:38.069710 17580 ts_tablet_manager.cc:1367] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:38.070153 17580 raft_consensus.cc:348] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.071241 17580 raft_consensus.cc:374] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.071353 17580 raft_consensus.cc:729] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.071554 17580 consensus_queue.cc:260] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.071659 17581 tablet_bootstrap.cc:654] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.072007 17580 ts_tablet_manager.cc:1398] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.072247 17580 tablet_bootstrap.cc:492] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:38.072979 17589 tablet_bootstrap.cc:492] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:38.073133 17580 tablet_bootstrap.cc:654] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.073987 17581 tablet_bootstrap.cc:492] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:38.074116 17581 ts_tablet_manager.cc:1367] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:09:38.074831 17589 tablet_bootstrap.cc:654] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.074857 17581 raft_consensus.cc:348] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.074988 17581 raft_consensus.cc:374] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.075042 17581 raft_consensus.cc:729] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.075202 17581 consensus_queue.cc:260] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.076732 17589 tablet_bootstrap.cc:492] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:38.076877 17589 ts_tablet_manager.cc:1367] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:09:38.077004 17584 tablet_bootstrap.cc:492] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.077376 17589 raft_consensus.cc:348] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.077566 17589 raft_consensus.cc:374] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.077642 17589 raft_consensus.cc:729] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.077828 17589 consensus_queue.cc:260] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.077879 16391 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.078800 17589 ts_tablet_manager.cc:1398] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:38.079092 17589 tablet_bootstrap.cc:492] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:38.079440 17582 tablet_bootstrap.cc:492] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:38.079540 17582 ts_tablet_manager.cc:1367] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:38.079720 17584 tablet_bootstrap.cc:654] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.080004 17582 raft_consensus.cc:348] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.080032 17589 tablet_bootstrap.cc:654] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.080102 17582 raft_consensus.cc:374] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.080135 17582 raft_consensus.cc:729] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.080247 17582 consensus_queue.cc:260] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.080504 17582 ts_tablet_manager.cc:1398] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.080684 17582 tablet_bootstrap.cc:492] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.081532 17582 tablet_bootstrap.cc:654] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.082309 17581 ts_tablet_manager.cc:1398] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:38.084236 17582 tablet_bootstrap.cc:492] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:38.084327 17582 ts_tablet_manager.cc:1367] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:38.084697 17583 tablet_bootstrap.cc:492] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:38.084733 17582 raft_consensus.cc:348] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.084790 17583 ts_tablet_manager.cc:1367] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:38.084817 17582 raft_consensus.cc:374] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.084847 17582 raft_consensus.cc:729] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.084949 17582 consensus_queue.cc:260] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.085166 17582 ts_tablet_manager.cc:1398] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.085197 17583 raft_consensus.cc:348] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.085304 17582 tablet_bootstrap.cc:492] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.085310 17583 raft_consensus.cc:374] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.085345 17583 raft_consensus.cc:729] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.085486 17583 consensus_queue.cc:260] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.086103 17582 tablet_bootstrap.cc:654] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.087754 17583 ts_tablet_manager.cc:1398] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.088526 17583 tablet_bootstrap.cc:492] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:38.088802 17584 tablet_bootstrap.cc:492] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:38.088941 17584 ts_tablet_manager.cc:1367] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:38.089447 17584 raft_consensus.cc:348] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.089586 17584 raft_consensus.cc:374] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.089622 17584 raft_consensus.cc:729] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.089754 17584 consensus_queue.cc:260] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.089457 17583 tablet_bootstrap.cc:654] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.091110 17585 tablet_bootstrap.cc:492] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:38.091220 17585 ts_tablet_manager.cc:1367] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.000s
I20240501 14:09:38.091432 17580 tablet_bootstrap.cc:492] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:38.091532 17580 ts_tablet_manager.cc:1367] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:38.091691 17584 ts_tablet_manager.cc:1398] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:38.091657 17585 raft_consensus.cc:348] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.091754 17585 raft_consensus.cc:374] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.091784 17585 raft_consensus.cc:729] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.091912 17585 consensus_queue.cc:260] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.092002 17580 raft_consensus.cc:348] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.092108 17580 raft_consensus.cc:374] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.092140 17580 raft_consensus.cc:729] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.092218 17585 ts_tablet_manager.cc:1398] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:38.092252 17580 consensus_queue.cc:260] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.092394 17582 tablet_bootstrap.cc:492] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:38.092471 17582 ts_tablet_manager.cc:1367] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:38.092531 17580 ts_tablet_manager.cc:1398] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.092897 17582 raft_consensus.cc:348] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.093008 17582 raft_consensus.cc:374] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.093057 17582 raft_consensus.cc:729] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.093173 17582 consensus_queue.cc:260] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.093444 17582 ts_tablet_manager.cc:1398] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.093773 12716 consensus_queue.cc:237] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.095500 12848 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.096138 12978 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.097275 17434 raft_consensus.cc:482] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.097368 17434 raft_consensus.cc:504] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.097560 17177 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.097687 17434 leader_election.cc:288] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:38.098202 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49fa7fe41af439688ef801bb2044485" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:38.098307 14957 raft_consensus.cc:2454] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:38.098392 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49fa7fe41af439688ef801bb2044485" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:38.098510 15090 raft_consensus.cc:2454] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:38.098577 15287 leader_election.cc:302] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.098718 17434 raft_consensus.cc:2781] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.098785 17434 raft_consensus.cc:482] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.098819 17434 raft_consensus.cc:3037] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.099483 15980 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.099839 17434 raft_consensus.cc:504] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.100075 15425 raft_consensus.cc:482] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.100142 17434 leader_election.cc:288] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:38.100152 15425 raft_consensus.cc:504] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.100473 15425 leader_election.cc:288] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:38.100492 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49fa7fe41af439688ef801bb2044485" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:38.100574 14957 raft_consensus.cc:3037] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.100901 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:38.101019 12716 raft_consensus.cc:2454] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:38.101222 15023 leader_election.cc:302] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:38.101425 14957 raft_consensus.cc:2454] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:38.101467 15422 raft_consensus.cc:2781] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.101519 15422 raft_consensus.cc:482] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.101567 15422 raft_consensus.cc:3037] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.101756 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:38.101806 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49fa7fe41af439688ef801bb2044485" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:38.101883 12585 raft_consensus.cc:2454] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:38.101897 15090 raft_consensus.cc:3037] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.102334 15287 leader_election.cc:302] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.102478 17434 raft_consensus.cc:2781] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.102478 15422 raft_consensus.cc:504] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.102540 17434 raft_consensus.cc:686] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:38.102643 17434 consensus_queue.cc:237] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.102849 15422 leader_election.cc:288] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:38.103703 17589 tablet_bootstrap.cc:492] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:38.103806 17589 ts_tablet_manager.cc:1367] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s
I20240501 14:09:38.104404 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:38.104501 12585 raft_consensus.cc:3037] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:38.104501 12652 consensus_peers.cc:485] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16324878552c4b6dbf10e8d5f73a0436. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.104629 12443 catalog_manager.cc:5484] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.105361 15090 raft_consensus.cc:2454] T b49fa7fe41af439688ef801bb2044485 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:38.105414 12585 raft_consensus.cc:2454] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:38.105782 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:38.105892 12716 raft_consensus.cc:3037] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.105973 17589 raft_consensus.cc:348] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.106069 17589 raft_consensus.cc:374] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.106093 17589 raft_consensus.cc:729] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.106199 17589 consensus_queue.cc:260] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.106889 12716 raft_consensus.cc:2454] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:38.107187 15023 leader_election.cc:302] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:38.107348 15425 raft_consensus.cc:2781] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.107420 15425 raft_consensus.cc:686] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:38.107431 17589 ts_tablet_manager.cc:1398] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240501 14:09:38.107535 15425 consensus_queue.cc:237] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:38.108700 17428 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.108907 17583 tablet_bootstrap.cc:492] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:38.108994 17583 ts_tablet_manager.cc:1367] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:38.109431 17583 raft_consensus.cc:348] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.109519 17583 raft_consensus.cc:374] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.109548 17583 raft_consensus.cc:729] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.109663 17583 consensus_queue.cc:260] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.112864 16724 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.114352 12439 catalog_manager.cc:5484] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.115736 17583 ts_tablet_manager.cc:1398] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:38.117162 15980 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.119218 12978 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.119221 12848 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.121796 12439 catalog_manager.cc:5484] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.130427 12848 consensus_queue.cc:237] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.131877 12716 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.131947 12978 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.132140 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 727a51852ac2428eb23f7690fec56a16. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.132301 17415 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.132472 16360 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.134382 16360 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.134949 12978 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.135008 12716 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.137398 12437 catalog_manager.cc:5484] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.145746 12848 consensus_queue.cc:237] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.147125 12978 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.147253 12585 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.147269 12785 consensus_peers.cc:485] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5648b7e0a1e744cea470feb7911d3d2c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.147557 17415 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.147719 16360 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.149390 17415 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.149751 12978 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.149952 12585 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.152297 12439 catalog_manager.cc:5484] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.155495 16523 raft_consensus.cc:482] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.155597 16523 raft_consensus.cc:504] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.156001 16523 leader_election.cc:288] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:38.156569 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a447cf75eb324846858ca40d152ea573" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:38.156708 15221 raft_consensus.cc:2454] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:38.157104 14891 leader_election.cc:302] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:38.157272 16523 raft_consensus.cc:2781] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.157341 16523 raft_consensus.cc:482] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.157368 16523 raft_consensus.cc:3037] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.158003 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a447cf75eb324846858ca40d152ea573" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:38.158129 12978 raft_consensus.cc:2454] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:38.158216 16523 raft_consensus.cc:504] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.158608 16523 leader_election.cc:288] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:38.158916 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a447cf75eb324846858ca40d152ea573" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:38.159031 15222 raft_consensus.cc:3037] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.159790 15222 raft_consensus.cc:2454] T a447cf75eb324846858ca40d152ea573 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:38.160058 14891 leader_election.cc:302] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:38.160215 16523 raft_consensus.cc:2781] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.160216 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a447cf75eb324846858ca40d152ea573" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:38.160315 16523 raft_consensus.cc:686] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:38.160320 12978 raft_consensus.cc:3037] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.160440 16523 consensus_queue.cc:237] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.161173 12978 raft_consensus.cc:2454] T a447cf75eb324846858ca40d152ea573 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:38.162173 12439 catalog_manager.cc:5484] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.167034 15090 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:38.167433 17480 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.173758 16523 raft_consensus.cc:482] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.173871 16523 raft_consensus.cc:504] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.174324 16523 leader_election.cc:288] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:38.174674 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cff740f14479a962d7c077bc53908" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:38.174860 12585 raft_consensus.cc:2454] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:38.175776 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cff740f14479a962d7c077bc53908" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:38.175905 15222 raft_consensus.cc:2454] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:38.176146 14893 leader_election.cc:302] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:38.176393 16523 raft_consensus.cc:2781] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.176457 16523 raft_consensus.cc:482] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.176508 16523 raft_consensus.cc:3037] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.177480 16523 raft_consensus.cc:504] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.177811 16523 leader_election.cc:288] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:38.178164 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cff740f14479a962d7c077bc53908" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:38.178264 12586 raft_consensus.cc:3037] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.178854 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095cff740f14479a962d7c077bc53908" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:38.178964 15222 raft_consensus.cc:3037] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.179128 12586 raft_consensus.cc:2454] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:38.179384 14893 leader_election.cc:302] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:38.179893 15222 raft_consensus.cc:2454] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:38.180835 16523 raft_consensus.cc:2781] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.180929 16523 raft_consensus.cc:686] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:38.181046 16523 consensus_queue.cc:237] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.182809 12439 catalog_manager.cc:5484] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.219355 12848 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:38.219970 17480 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.226565 12716 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:38.227030 17481 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.231789 12585 consensus_queue.cc:237] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:38.232903 15090 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:38.233219 12848 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:38.233590 17272 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.234426 17272 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.236204 17272 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.236681 15090 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.239466 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:38.239807 12442 catalog_manager.cc:5484] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.240403 16360 raft_consensus.cc:1050] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:38.240785 16360 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:38.242091 15353 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:38.242602 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:38.243835 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:38.244074 16882 tablet_replica.cc:330] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:38.244226 16882 raft_consensus.cc:2229] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.244364 16882 raft_consensus.cc:2258] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.245100 16882 ts_tablet_manager.cc:1875] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:38.245445 12978 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:38.245836 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:38.246986 12848 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.247422 16882 ts_tablet_manager.cc:1888] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:38.247480 16882 log.cc:1196] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/515d8f8e9d744aa4b2c8a4090c2f72e8
I20240501 14:09:38.247913 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 (table rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]) successfully deleted
I20240501 14:09:38.250250 15090 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:38.250572 15878 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:38.255733 17415 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.257673 12978 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.257716 15353 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.260836 12441 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 7 to 8, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.263041 15090 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.267100 12848 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:38.269047 15090 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:38.269443 16360 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:38.270002 15351 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:38.270722 16360 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:38.271787 16360 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.273038 15351 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.273335 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:38.273869 15090 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:38.274528 12445 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.276935 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:38.277156 16883 tablet_replica.cc:330] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:38.277310 16883 raft_consensus.cc:2229] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.277496 16883 raft_consensus.cc:2258] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.278172 16883 ts_tablet_manager.cc:1875] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:38.278921 15353 raft_consensus.cc:1203] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:38.280754 16883 ts_tablet_manager.cc:1888] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:38.280829 16883 log.cc:1196] T bd5489eab5b04751bc8b3e5fb9e8cf73 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/bd5489eab5b04751bc8b3e5fb9e8cf73
I20240501 14:09:38.281706 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet bd5489eab5b04751bc8b3e5fb9e8cf73 (table rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]) successfully deleted
I20240501 14:09:38.289244 12827 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:09:38.289599 15200 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:09:38.293582 14937 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.300976 15333 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:09:38.306077 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:38.306942 12566 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.307193 12958 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:09:38.310249 12585 raft_consensus.cc:1261] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.310691 16523 consensus_queue.cc:1035] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.311558 15222 raft_consensus.cc:1261] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.311980 16523 consensus_queue.cc:1035] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.315157 17406 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.315418 12442 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]). Expected version 1 got 0
I20240501 14:09:38.315452 17406 tablet.cc:1620] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:38.316550 12439 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]). Expected version 1 got 0
I20240501 14:09:38.322338 12442 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]). Expected version 1 got 0
I20240501 14:09:38.323837 12698 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 14:09:38.327694 15222 raft_consensus.cc:1203] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:38.330807 17687 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.331130 17687 tablet.cc:1620] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:38.331247 12445 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9]). Expected version 1 got 0
I20240501 14:09:38.331492 17688 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:38.331785 17688 tablet.cc:1620] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.333302 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9] alter complete (version 1)
I20240501 14:09:38.343910 15068 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:09:38.394474 14957 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:38.396129 12848 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.396375 15090 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.396608 16771 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:38.396811 14891 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e -> Peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): Couldn't send request to peer e6ab42bc38694f7f9fc8bfcc0a1483b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df57dee35f924801a23bd9b5c547a55f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.396855 15895 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.398471 15421 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:38.399667 15090 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:38.401641 12439 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.403281 12848 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:38.406915 17697 ts_tablet_manager.cc:902] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:38.407441 17697 tablet_copy_client.cc:308] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:38.408533 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet d178c3291efc44369e3adb921ccc6517 from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:34877)
I20240501 14:09:38.408622 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet d178c3291efc44369e3adb921ccc6517 from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:34877: session id = f23f426391724d6688aafffe7ee017d8-d178c3291efc44369e3adb921ccc6517
I20240501 14:09:38.409430 12607 tablet_copy_source_session.cc:205] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.410568 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d178c3291efc44369e3adb921ccc6517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.412768 17697 tablet_copy_client.cc:783] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.412995 17697 tablet_copy_client.cc:647] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.413120 12586 consensus_queue.cc:237] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.414184 17697 tablet_copy_client.cc:523] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.414680 12978 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.414790 12848 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.415446 12521 consensus_peers.cc:485] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb336cffd30b447b880af97da6882d27. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.415534 17481 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.415654 17697 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.415700 17480 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.418423 17480 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.420300 12848 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.421422 12439 catalog_manager.cc:5484] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.424278 12978 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.435111 15090 consensus_queue.cc:237] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
W20240501 14:09:38.437098 15025 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d521005dcbf74e4fb9a67fe9992ce540. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.437098 15351 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.437294 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:38.438050 15222 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.438380 17717 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 14:09:38.439605 15422 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.441618 17717 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.442222 15221 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.444943 17406 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.445025 12445 catalog_manager.cc:5484] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.445276 17406 tablet.cc:1620] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:38.447185 15351 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.452059 17688 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.452384 17688 tablet.cc:1620] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:38.456156 17687 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:38.456480 17687 tablet.cc:1620] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:38.459481 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=7b3c1d32f923490da5973abe5496b4e9] alter complete (version 2)
I20240501 14:09:38.461009 17697 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.461375 17697 tablet_bootstrap.cc:492] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:38.461555 17697 ts_tablet_manager.cc:1367] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.046s user 0.025s sys 0.002s
I20240501 14:09:38.461984 17697 raft_consensus.cc:348] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.462009 12586 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.462170 17697 raft_consensus.cc:729] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:38.462280 17697 consensus_queue.cc:260] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.463156 17697 ts_tablet_manager.cc:1398] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.463354 15222 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:38.463572 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-d178c3291efc44369e3adb921ccc6517 received from {username='slave'} at 127.6.82.8:34877
I20240501 14:09:38.463637 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session f23f426391724d6688aafffe7ee017d8-d178c3291efc44369e3adb921ccc6517 on tablet d178c3291efc44369e3adb921ccc6517 with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:38.463800 15655 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.463929 12848 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:38.464234 12523 consensus_peers.cc:485] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7693e3e204634676a758726bb6d3d2ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.464430 15655 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.467065 17481 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.467532 15222 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.469506 12848 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.469934 12442 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.478441 12716 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.479818 15090 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:38.480038 12652 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68b04992f1f94b36b01dfde24a42318d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.480051 12978 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:38.480401 16305 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.480564 16837 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:38.482272 15980 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.482692 15090 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.482702 12978 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.485357 12439 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.493291 17738 ts_tablet_manager.cc:902] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:38.493894 17738 tablet_copy_client.cc:308] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:38.494338 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 16324878552c4b6dbf10e8d5f73a0436 from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:37435)
I20240501 14:09:38.494460 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 16324878552c4b6dbf10e8d5f73a0436 from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:37435: session id = 8a13c028ba8a4a659b74af6c265df95e-16324878552c4b6dbf10e8d5f73a0436
I20240501 14:09:38.495347 12738 tablet_copy_source_session.cc:205] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.495407 12978 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:38.496229 17738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16324878552c4b6dbf10e8d5f73a0436. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:09:38.497049 12913 consensus_peers.cc:485] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 188ce319682147ddaad30aae7232a311. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.497042 15090 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.497443 12848 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.497473 16391 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.497947 17167 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.498425 17738 tablet_copy_client.cc:783] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.498610 17738 tablet_copy_client.cc:647] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.499699 17738 tablet_copy_client.cc:523] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.499967 16391 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:38.500396 15090 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:38.501116 17738 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.503330 12442 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.504323 12848 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:38.507782 17751 ts_tablet_manager.cc:902] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:38.515110 17751 tablet_copy_client.cc:308] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:38.515672 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 444c1a9d0e6943dc880161ae320673b6 from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:51545)
I20240501 14:09:38.515790 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 444c1a9d0e6943dc880161ae320673b6 from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:51545: session id = 66e2dc34330848c3b8c2dd15898d34c8-444c1a9d0e6943dc880161ae320673b6
I20240501 14:09:38.516803 12607 tablet_copy_source_session.cc:205] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.517794 17434 consensus_queue.cc:1035] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.518922 17751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1a9d0e6943dc880161ae320673b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.521257 17751 tablet_copy_client.cc:783] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.521485 17751 tablet_copy_client.cc:647] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.523159 17751 tablet_copy_client.cc:523] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.524559 16475 consensus_queue.cc:1035] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.525120 17751 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:38.532311 17738 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.532603 17738 tablet_bootstrap.cc:492] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:38.532810 17738 ts_tablet_manager.cc:1367] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.005s
I20240501 14:09:38.533679 17738 raft_consensus.cc:348] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.533804 17738 raft_consensus.cc:729] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:38.533905 17738 consensus_queue.cc:260] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.534746 17738 ts_tablet_manager.cc:1398] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.535512 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-16324878552c4b6dbf10e8d5f73a0436 received from {username='slave'} at 127.6.82.5:37435
I20240501 14:09:38.535600 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-16324878552c4b6dbf10e8d5f73a0436 on tablet 16324878552c4b6dbf10e8d5f73a0436 with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:38.537776 17771 ts_tablet_manager.cc:902] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:38.538309 17771 tablet_copy_client.cc:308] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:38.538880 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet a5957da39df64253a2b8d7f388308719 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:51695)
I20240501 14:09:38.539011 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet a5957da39df64253a2b8d7f388308719 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:51695: session id = ab90e77f0027449bb3155980ca95593c-a5957da39df64253a2b8d7f388308719
I20240501 14:09:38.539767 12869 tablet_copy_source_session.cc:205] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.540721 17771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5957da39df64253a2b8d7f388308719. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.542101 17781 ts_tablet_manager.cc:902] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:38.542663 17781 tablet_copy_client.cc:308] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:38.543015 17771 tablet_copy_client.cc:783] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.543112 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet 727a51852ac2428eb23f7690fec56a16 from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:51695)
I20240501 14:09:38.543170 17771 tablet_copy_client.cc:647] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.543229 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet 727a51852ac2428eb23f7690fec56a16 from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:51695: session id = ab90e77f0027449bb3155980ca95593c-727a51852ac2428eb23f7690fec56a16
I20240501 14:09:38.544154 12869 tablet_copy_source_session.cc:205] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.544225 17771 tablet_copy_client.cc:523] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.545441 17781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727a51852ac2428eb23f7690fec56a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.545738 17771 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:38.546643 17717 consensus_queue.cc:1035] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.549429 17781 tablet_copy_client.cc:783] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.549584 17781 tablet_copy_client.cc:647] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.550659 17781 tablet_copy_client.cc:523] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.551355 17697 ts_tablet_manager.cc:902] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:38.551895 17697 tablet_copy_client.cc:308] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:38.552086 17781 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:38.552386 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet 5648b7e0a1e744cea470feb7911d3d2c from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:58949)
I20240501 14:09:38.552485 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet 5648b7e0a1e744cea470feb7911d3d2c from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:58949: session id = f23f426391724d6688aafffe7ee017d8-5648b7e0a1e744cea470feb7911d3d2c
I20240501 14:09:38.553472 12869 tablet_copy_source_session.cc:205] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.555548 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5648b7e0a1e744cea470feb7911d3d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.557327 17697 tablet_copy_client.cc:783] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.557441 17697 tablet_copy_client.cc:647] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.559150 17697 tablet_copy_client.cc:523] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.560333 17697 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.562773 17751 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.562919 15425 consensus_queue.cc:1035] T 79e8de8cd4d2407c99ea4c5520d007f5 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.563287 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:38.565177 17751 tablet_bootstrap.cc:492] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:38.565371 17751 ts_tablet_manager.cc:1367] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.000s
I20240501 14:09:38.567065 17751 raft_consensus.cc:348] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:38.567188 17751 raft_consensus.cc:729] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:38.567291 17751 consensus_queue.cc:260] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:38.567605 17751 ts_tablet_manager.cc:1398] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.568990 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-444c1a9d0e6943dc880161ae320673b6 received from {username='slave'} at 127.6.82.7:51545
I20240501 14:09:38.569059 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-444c1a9d0e6943dc880161ae320673b6 on tablet 444c1a9d0e6943dc880161ae320673b6 with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:38.572103 17406 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:38.575675 17687 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:38.578195 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=7b3c1d32f923490da5973abe5496b4e9] alter complete (version 3)
I20240501 14:09:38.582278 17688 tablet.cc:1722] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:38.598250 17697 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.598681 17697 tablet_bootstrap.cc:492] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:38.598927 17697 ts_tablet_manager.cc:1367] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.001s
I20240501 14:09:38.599534 17697 raft_consensus.cc:348] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.599758 17697 raft_consensus.cc:729] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:38.599829 17771 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.599963 17697 consensus_queue.cc:260] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.600111 17771 tablet_bootstrap.cc:492] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:38.600178 17781 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.600291 17771 ts_tablet_manager.cc:1367] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.004s
I20240501 14:09:38.600425 17781 tablet_bootstrap.cc:492] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:38.600546 17781 ts_tablet_manager.cc:1367] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.049s user 0.026s sys 0.000s
I20240501 14:09:38.600844 17781 raft_consensus.cc:348] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.600947 17697 ts_tablet_manager.cc:1398] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.601420 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-5648b7e0a1e744cea470feb7911d3d2c received from {username='slave'} at 127.6.82.8:58949
I20240501 14:09:38.601491 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session f23f426391724d6688aafffe7ee017d8-5648b7e0a1e744cea470feb7911d3d2c on tablet 5648b7e0a1e744cea470feb7911d3d2c with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:38.601693 17781 raft_consensus.cc:729] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:38.601020 17771 raft_consensus.cc:348] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.601806 17771 raft_consensus.cc:729] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:38.601907 17771 consensus_queue.cc:260] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.601821 17781 consensus_queue.cc:260] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.603363 17781 ts_tablet_manager.cc:1398] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:38.603852 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-727a51852ac2428eb23f7690fec56a16 received from {username='slave'} at 127.6.82.6:51695
I20240501 14:09:38.603950 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session ab90e77f0027449bb3155980ca95593c-727a51852ac2428eb23f7690fec56a16 on tablet 727a51852ac2428eb23f7690fec56a16 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:38.606124 17771 ts_tablet_manager.cc:1398] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:38.607035 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-a5957da39df64253a2b8d7f388308719 received from {username='slave'} at 127.6.82.6:51695
I20240501 14:09:38.607117 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session ab90e77f0027449bb3155980ca95593c-a5957da39df64253a2b8d7f388308719 on tablet a5957da39df64253a2b8d7f388308719 with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:38.645797 16305 raft_consensus.cc:1050] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:38.646221 16305 consensus_queue.cc:237] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:38.647650 12848 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.648058 12978 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.648034 16837 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.648085 15351 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.648753 17177 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.649055 17177 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.650800 17177 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.651892 12978 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.653597 12848 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.654057 12439 catalog_manager.cc:5484] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.655894 15351 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.659011 12716 consensus_queue.cc:237] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:38.660287 12978 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:38.660537 15351 raft_consensus.cc:1261] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:38.660699 16305 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.660897 16837 consensus_queue.cc:1035] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.663156 12978 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.663703 16837 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.665033 15353 raft_consensus.cc:2932] T 1f1c8ff4149c42aa9517af442c38060c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:38.665997 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f1c8ff4149c42aa9517af442c38060c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:38.666538 12440 catalog_manager.cc:5484] T 1f1c8ff4149c42aa9517af442c38060c P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.668160 15895 consensus_queue.cc:1035] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.670863 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1c8ff4149c42aa9517af442c38060c with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:38.671350 16523 consensus_queue.cc:1035] T a447cf75eb324846858ca40d152ea573 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:38.674072 17850 tablet_replica.cc:330] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:38.674309 17850 raft_consensus.cc:2229] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.674618 17850 raft_consensus.cc:2258] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.675354 17850 ts_tablet_manager.cc:1875] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:38.676764 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:38.678385 17850 ts_tablet_manager.cc:1888] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:38.678463 17850 log.cc:1196] T 1f1c8ff4149c42aa9517af442c38060c P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/1f1c8ff4149c42aa9517af442c38060c
I20240501 14:09:38.679047 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 1f1c8ff4149c42aa9517af442c38060c (table rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]) successfully deleted
I20240501 14:09:38.681144 12440 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:38.681259 12440 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:38.683315 12440 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 095cff740f14479a962d7c077bc53908
I20240501 14:09:38.683993 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 095cff740f14479a962d7c077bc53908 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:38 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:38.684037 15200 tablet_service.cc:1514] Processing DeleteTablet for tablet 095cff740f14479a962d7c077bc53908 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:38 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:38.684309 17863 tablet_replica.cc:330] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:38.684376 17864 tablet_replica.cc:330] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:38.684536 17864 raft_consensus.cc:2229] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.684568 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 095cff740f14479a962d7c077bc53908 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:38 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:38.684705 17864 raft_consensus.cc:2258] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.684854 17863 raft_consensus.cc:2229] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:38.685508 17863 raft_consensus.cc:2258] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.685870 15200 tablet_service.cc:1467] Processing CreateTablet for tablet db946894cd2348b0be6b8bb574c19667 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.686018 17865 tablet_replica.cc:330] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:38.686128 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db946894cd2348b0be6b8bb574c19667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.686146 17865 raft_consensus.cc:2229] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.686264 17865 raft_consensus.cc:2258] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.686681 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 26b93a7b5dd24262a9e70e407653ed50 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.686795 12566 tablet_service.cc:1467] Processing CreateTablet for tablet a74a465ca78c43419b2066dcf077a691 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.686988 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b93a7b5dd24262a9e70e407653ed50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.687047 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74a465ca78c43419b2066dcf077a691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.687067 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 26b93a7b5dd24262a9e70e407653ed50 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.687306 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b93a7b5dd24262a9e70e407653ed50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.688632 17864 ts_tablet_manager.cc:1875] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:38.689807 14937 tablet_service.cc:1467] Processing CreateTablet for tablet db946894cd2348b0be6b8bb574c19667 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.690070 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db946894cd2348b0be6b8bb574c19667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.690158 17865 ts_tablet_manager.cc:1875] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:38.691191 17863 ts_tablet_manager.cc:1875] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:38.692179 17864 ts_tablet_manager.cc:1888] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:38.692248 17864 log.cc:1196] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/095cff740f14479a962d7c077bc53908
I20240501 14:09:38.692523 17864 ts_tablet_manager.cc:1909] T 095cff740f14479a962d7c077bc53908 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:38.692584 17865 ts_tablet_manager.cc:1888] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:38.692706 17865 log.cc:1196] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/095cff740f14479a962d7c077bc53908
I20240501 14:09:38.693186 17865 ts_tablet_manager.cc:1909] T 095cff740f14479a962d7c077bc53908 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:38.693329 17868 tablet_bootstrap.cc:492] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:38.693305 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table_new_name [id=7b3c1d32f923490da5973abe5496b4e9]) successfully deleted
I20240501 14:09:38.693375 12827 tablet_service.cc:1467] Processing CreateTablet for tablet a74a465ca78c43419b2066dcf077a691 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.693707 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74a465ca78c43419b2066dcf077a691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.693850 17863 ts_tablet_manager.cc:1888] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:38.693907 17863 log.cc:1196] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/095cff740f14479a962d7c077bc53908
I20240501 14:09:38.693871 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table_new_name [id=7b3c1d32f923490da5973abe5496b4e9]) successfully deleted
I20240501 14:09:38.694161 17863 ts_tablet_manager.cc:1909] T 095cff740f14479a962d7c077bc53908 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:38.694454 17868 tablet_bootstrap.cc:654] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.694896 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 095cff740f14479a962d7c077bc53908 (table rebalancer_dynamic_table_new_name [id=7b3c1d32f923490da5973abe5496b4e9]) successfully deleted
I20240501 14:09:38.695472 17868 tablet_bootstrap.cc:492] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:38.695466 17867 tablet_bootstrap.cc:492] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:38.695541 17868 ts_tablet_manager.cc:1367] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:38.695982 17868 raft_consensus.cc:348] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.696090 17868 raft_consensus.cc:374] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.696128 17868 raft_consensus.cc:729] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.696255 17868 consensus_queue.cc:260] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.696378 17866 tablet_bootstrap.cc:492] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:38.696396 15331 tablet_service.cc:1467] Processing CreateTablet for tablet a74a465ca78c43419b2066dcf077a691 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.696533 17868 ts_tablet_manager.cc:1398] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.696668 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74a465ca78c43419b2066dcf077a691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.697111 15333 tablet_service.cc:1467] Processing CreateTablet for tablet db946894cd2348b0be6b8bb574c19667 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.697306 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db946894cd2348b0be6b8bb574c19667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.697739 17869 tablet_bootstrap.cc:492] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:38.698459 17869 tablet_bootstrap.cc:654] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.698859 17873 tablet_bootstrap.cc:492] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.698954 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 26b93a7b5dd24262a9e70e407653ed50 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=5d6072b135664fb389f8361676e6c97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:38.699182 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b93a7b5dd24262a9e70e407653ed50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.699640 17867 tablet_bootstrap.cc:654] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.701730 17873 tablet_bootstrap.cc:654] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.702394 17866 tablet_bootstrap.cc:654] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.702865 17873 tablet_bootstrap.cc:492] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:38.702960 17873 ts_tablet_manager.cc:1367] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:38.703445 17873 raft_consensus.cc:348] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.703557 17873 raft_consensus.cc:374] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.703593 17873 raft_consensus.cc:729] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.703724 17873 consensus_queue.cc:260] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.703881 17870 tablet_bootstrap.cc:492] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.704028 17873 ts_tablet_manager.cc:1398] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:38.704170 17869 tablet_bootstrap.cc:492] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:38.704188 17873 tablet_bootstrap.cc:492] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.704263 17869 ts_tablet_manager.cc:1367] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:38.704890 17870 tablet_bootstrap.cc:654] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.705001 17873 tablet_bootstrap.cc:654] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.705583 17869 raft_consensus.cc:348] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.705910 17869 raft_consensus.cc:374] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.706013 17869 raft_consensus.cc:729] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.706207 17872 tablet_bootstrap.cc:492] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:38.706218 17869 consensus_queue.cc:260] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.707202 17872 tablet_bootstrap.cc:654] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.708561 17869 ts_tablet_manager.cc:1398] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:38.709210 17867 tablet_bootstrap.cc:492] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:38.709534 17867 ts_tablet_manager.cc:1367] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:38.710086 17867 raft_consensus.cc:348] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.710201 17867 raft_consensus.cc:374] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.710228 17867 raft_consensus.cc:729] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.710419 17167 raft_consensus.cc:482] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.710501 17167 raft_consensus.cc:504] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.710927 17167 leader_election.cc:288] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:38.711483 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b93a7b5dd24262a9e70e407653ed50" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:38.711565 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b93a7b5dd24262a9e70e407653ed50" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:38.711755 17867 consensus_queue.cc:260] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
W20240501 14:09:38.711889 12916 leader_election.cc:341] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:38.711990 12716 raft_consensus.cc:2454] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:38.712517 12913 leader_election.cc:302] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: f23f426391724d6688aafffe7ee017d8
I20240501 14:09:38.712703 17167 raft_consensus.cc:2781] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.712769 17167 raft_consensus.cc:482] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.712806 17167 raft_consensus.cc:3037] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.713420 17872 tablet_bootstrap.cc:492] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:38.713511 17872 ts_tablet_manager.cc:1367] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:38.713863 17167 raft_consensus.cc:504] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.714242 17167 leader_election.cc:288] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:38.714061 17872 raft_consensus.cc:348] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.714411 17872 raft_consensus.cc:374] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.714457 17872 raft_consensus.cc:729] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.714486 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b93a7b5dd24262a9e70e407653ed50" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:38.714629 17872 consensus_queue.cc:260] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
W20240501 14:09:38.714849 12916 leader_election.cc:341] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:38.714937 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b93a7b5dd24262a9e70e407653ed50" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:38.715027 12716 raft_consensus.cc:3037] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.715718 17872 ts_tablet_manager.cc:1398] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:38.715976 12716 raft_consensus.cc:2454] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:38.716253 12913 leader_election.cc:302] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: f23f426391724d6688aafffe7ee017d8
I20240501 14:09:38.717489 17167 raft_consensus.cc:2781] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.717581 17167 raft_consensus.cc:686] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:38.717703 17167 consensus_queue.cc:237] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.719229 17867 ts_tablet_manager.cc:1398] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 14:09:38.719810 12440 catalog_manager.cc:5484] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.720113 17866 tablet_bootstrap.cc:492] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:38.720221 17866 ts_tablet_manager.cc:1367] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 14:09:38.720679 17866 raft_consensus.cc:348] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.720789 17866 raft_consensus.cc:374] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.720820 17866 raft_consensus.cc:729] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.720943 17866 consensus_queue.cc:260] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.721228 17866 ts_tablet_manager.cc:1398] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.722332 17870 tablet_bootstrap.cc:492] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:38.722445 17870 ts_tablet_manager.cc:1367] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:38.722832 17873 tablet_bootstrap.cc:492] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:38.722931 17873 ts_tablet_manager.cc:1367] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240501 14:09:38.722976 17870 raft_consensus.cc:348] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.723084 17870 raft_consensus.cc:374] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.723117 17870 raft_consensus.cc:729] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.723278 17870 consensus_queue.cc:260] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.723400 17873 raft_consensus.cc:348] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.723510 17873 raft_consensus.cc:374] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.723541 17873 raft_consensus.cc:729] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.723598 17870 ts_tablet_manager.cc:1398] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.723663 17873 consensus_queue.cc:260] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.723949 17873 ts_tablet_manager.cc:1398] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.724128 17873 tablet_bootstrap.cc:492] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.725498 17873 tablet_bootstrap.cc:654] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.727694 17873 tablet_bootstrap.cc:492] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:38.727838 17873 ts_tablet_manager.cc:1367] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:09:38.728482 17873 raft_consensus.cc:348] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.728575 17873 raft_consensus.cc:374] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.728612 17873 raft_consensus.cc:729] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.728750 17873 consensus_queue.cc:260] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:38.729099 17873 ts_tablet_manager.cc:1398] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.734552 12716 consensus_queue.cc:237] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:38.735953 12849 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.736107 12978 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.736523 16837 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.736706 16305 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:09:38.736884 12653 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0e8e476bc46420d992a43bf3b4c73fd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.738606 16305 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.739151 12978 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.740783 17434 raft_consensus.cc:482] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.740885 17434 raft_consensus.cc:504] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.741346 17434 leader_election.cc:288] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:38.741657 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74a465ca78c43419b2066dcf077a691" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:38.741801 12848 raft_consensus.cc:2454] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:38.742038 12442 catalog_manager.cc:5484] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.742255 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74a465ca78c43419b2066dcf077a691" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:38.742377 12586 raft_consensus.cc:2454] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:38.742697 15288 leader_election.cc:302] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.742836 12849 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:38.743288 17434 raft_consensus.cc:2781] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.743345 17434 raft_consensus.cc:482] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.743379 17434 raft_consensus.cc:3037] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.744381 17434 raft_consensus.cc:504] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.744704 17434 leader_election.cc:288] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:38.745555 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74a465ca78c43419b2066dcf077a691" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:38.745640 12849 raft_consensus.cc:3037] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.746330 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74a465ca78c43419b2066dcf077a691" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:38.746470 12586 raft_consensus.cc:3037] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.746492 12849 raft_consensus.cc:2454] T a74a465ca78c43419b2066dcf077a691 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:38.746857 15289 leader_election.cc:302] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.747066 17434 raft_consensus.cc:2781] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.747135 17434 raft_consensus.cc:686] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:38.747249 17434 consensus_queue.cc:237] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:38.747538 12586 raft_consensus.cc:2454] T a74a465ca78c43419b2066dcf077a691 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:38.749400 12439 catalog_manager.cc:5484] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.751137 12716 consensus_queue.cc:237] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:38.752518 15222 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.752647 14957 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:38.752795 12653 consensus_peers.cc:485] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee9a1e9b13f546cc81aae6d729c8974f. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.752977 17177 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.753136 16305 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.755053 16305 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.755512 14957 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.755601 15222 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.758338 12439 catalog_manager.cc:5484] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.764297 17255 raft_consensus.cc:482] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.764413 17255 raft_consensus.cc:504] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.764793 17255 leader_election.cc:288] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:38.765113 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db946894cd2348b0be6b8bb574c19667" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:38.765216 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db946894cd2348b0be6b8bb574c19667" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:38.765259 15353 raft_consensus.cc:2454] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:38.765362 14957 raft_consensus.cc:2454] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:38.765563 15157 leader_election.cc:302] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.765707 17255 raft_consensus.cc:2781] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.765761 17255 raft_consensus.cc:482] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.765798 17255 raft_consensus.cc:3037] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.766480 12716 consensus_queue.cc:237] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:38.766688 17255 raft_consensus.cc:504] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.767045 17255 leader_election.cc:288] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:38.767375 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db946894cd2348b0be6b8bb574c19667" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:38.767491 15353 raft_consensus.cc:3037] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.767881 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db946894cd2348b0be6b8bb574c19667" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
W20240501 14:09:38.767922 12654 consensus_peers.cc:485] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6628290de33e4aba8ccd1651d1d054dd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.767974 14957 raft_consensus.cc:3037] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.768023 12849 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.768488 15353 raft_consensus.cc:2454] T db946894cd2348b0be6b8bb574c19667 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:38.768985 14957 raft_consensus.cc:2454] T db946894cd2348b0be6b8bb574c19667 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:38.769157 12978 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.769155 15980 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.768774 15157 leader_election.cc:302] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.769739 15980 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.770349 17255 raft_consensus.cc:2781] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.770439 17255 raft_consensus.cc:686] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:38.770581 17255 consensus_queue.cc:237] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:38.771034 15980 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.771755 12978 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.772997 12438 catalog_manager.cc:5484] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.773190 12849 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.775087 12439 catalog_manager.cc:5484] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 4, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:38.784319 15353 consensus_queue.cc:237] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:38.785701 14957 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:38.785887 15090 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:38.786047 15288 consensus_peers.cc:485] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 565e6fb440dc4b00b3e2d0e1936889c8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.786077 17434 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.786578 17434 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:38.787770 17925 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.788568 15090 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.791098 12442 catalog_manager.cc:5484] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.791965 14957 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:09:38.794386 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:38.800433 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef85be81b074f65b7c022b03061064a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3]), partition=
I20240501 14:09:38.800714 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef85be81b074f65b7c022b03061064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.802546 17938 tablet_bootstrap.cc:492] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:38.803105 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef85be81b074f65b7c022b03061064a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3]), partition=
I20240501 14:09:38.803367 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef85be81b074f65b7c022b03061064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.803575 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef85be81b074f65b7c022b03061064a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3]), partition=
I20240501 14:09:38.803838 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef85be81b074f65b7c022b03061064a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.806085 17873 tablet_bootstrap.cc:492] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:38.806248 17870 tablet_bootstrap.cc:492] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.807058 17873 tablet_bootstrap.cc:654] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.807180 17870 tablet_bootstrap.cc:654] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.808773 17938 tablet_bootstrap.cc:654] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:38.809989 17873 tablet_bootstrap.cc:492] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:38.810083 17873 ts_tablet_manager.cc:1367] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:38.810513 17873 raft_consensus.cc:348] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.810621 17873 raft_consensus.cc:374] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.810655 17873 raft_consensus.cc:729] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.810808 17873 consensus_queue.cc:260] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.811084 17873 ts_tablet_manager.cc:1398] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:38.812199 17938 tablet_bootstrap.cc:492] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:38.812355 17938 ts_tablet_manager.cc:1367] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:38.813306 17938 raft_consensus.cc:348] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.813493 17938 raft_consensus.cc:374] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.813584 17938 raft_consensus.cc:729] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.813794 17938 consensus_queue.cc:260] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.814213 17938 ts_tablet_manager.cc:1398] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:38.819101 17870 tablet_bootstrap.cc:492] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:38.819207 17870 ts_tablet_manager.cc:1367] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:38.819669 17870 raft_consensus.cc:348] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.819761 17870 raft_consensus.cc:374] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:38.819787 17870 raft_consensus.cc:729] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:38.819905 17870 consensus_queue.cc:260] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.820192 17870 ts_tablet_manager.cc:1398] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.911110 17177 raft_consensus.cc:1050] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:38.911610 17177 consensus_queue.cc:237] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:38.913833 12849 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.914245 12586 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.914240 15980 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.915066 15980 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.916693 15222 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:38.917127 17177 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:38.919173 15980 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.921196 17738 ts_tablet_manager.cc:902] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:38.921749 17738 tablet_copy_client.cc:308] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:38.923398 12441 catalog_manager.cc:5484] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:38.923691 12586 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.926467 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 68b04992f1f94b36b01dfde24a42318d from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:37435)
I20240501 14:09:38.926568 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 68b04992f1f94b36b01dfde24a42318d from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:37435: session id = 8a13c028ba8a4a659b74af6c265df95e-68b04992f1f94b36b01dfde24a42318d
I20240501 14:09:38.926621 18010 ts_tablet_manager.cc:902] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Initiating tablet copy from peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:38.927793 18010 tablet_copy_client.cc:308] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.5:33177
I20240501 14:09:38.928386 14978 tablet_copy_service.cc:140] P 8a13c028ba8a4a659b74af6c265df95e: Received BeginTabletCopySession request for tablet df57dee35f924801a23bd9b5c547a55f from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 ({username='slave'} at 127.6.82.2:40159)
I20240501 14:09:38.928507 14978 tablet_copy_service.cc:161] P 8a13c028ba8a4a659b74af6c265df95e: Beginning new tablet copy session on tablet df57dee35f924801a23bd9b5c547a55f from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 at {username='slave'} at 127.6.82.2:40159: session id = e6ab42bc38694f7f9fc8bfcc0a1483b7-df57dee35f924801a23bd9b5c547a55f
I20240501 14:09:38.929687 14978 tablet_copy_source_session.cc:205] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.930655 18010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df57dee35f924801a23bd9b5c547a55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.931214 14957 raft_consensus.cc:1203] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:38.932716 18010 tablet_copy_client.cc:783] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.932868 18010 tablet_copy_client.cc:647] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.932912 15221 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.934118 12738 tablet_copy_source_session.cc:205] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:38.935599 17738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68b04992f1f94b36b01dfde24a42318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:38.937965 17738 tablet_copy_client.cc:783] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:38.938098 17738 tablet_copy_client.cc:647] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:38.942107 12716 consensus_queue.cc:237] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:38.943655 12586 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:38.944258 16837 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.944936 17738 tablet_copy_client.cc:523] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.944934 15222 raft_consensus.cc:1261] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:38.945704 18010 tablet_copy_client.cc:523] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:38.946084 15980 consensus_queue.cc:1035] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:38.946796 16837 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.947263 18010 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:38.947263 17738 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:38.947729 12586 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.950143 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40b29fb993f34e99ad3747a2ec7d6c4f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:38.951643 12439 catalog_manager.cc:5484] T 40b29fb993f34e99ad3747a2ec7d6c4f P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.954361 15222 raft_consensus.cc:2932] T 40b29fb993f34e99ad3747a2ec7d6c4f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:38.955840 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 40b29fb993f34e99ad3747a2ec7d6c4f with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:38.956542 17850 tablet_replica.cc:330] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:38.956692 17850 raft_consensus.cc:2229] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:38.956833 17850 pending_rounds.cc:62] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Trying to abort 1 pending ops.
I20240501 14:09:38.956964 18010 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.956897 17850 pending_rounds.cc:68] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022889283217944576 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "40b29fb993f34e99ad3747a2ec7d6c4f" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } } }
I20240501 14:09:38.957082 17850 raft_consensus.cc:2894] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:09:38.957151 17850 raft_consensus.cc:2866] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:38.957206 17850 raft_consensus.cc:2258] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:38.957226 18010 tablet_bootstrap.cc:492] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap complete.
I20240501 14:09:38.957397 18010 ts_tablet_manager.cc:1367] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20240501 14:09:38.957988 18010 raft_consensus.cc:348] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:38.958106 18010 raft_consensus.cc:729] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: LEARNER
I20240501 14:09:38.958205 18010 consensus_queue.cc:260] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:38.958487 18010 ts_tablet_manager.cc:1398] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:38.958521 17850 ts_tablet_manager.cc:1875] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:38.959059 14978 tablet_copy_service.cc:342] P 8a13c028ba8a4a659b74af6c265df95e: Request end of tablet copy session e6ab42bc38694f7f9fc8bfcc0a1483b7-df57dee35f924801a23bd9b5c547a55f received from {username='slave'} at 127.6.82.2:40159
I20240501 14:09:38.959141 14978 tablet_copy_service.cc:433] P 8a13c028ba8a4a659b74af6c265df95e: ending tablet copy session e6ab42bc38694f7f9fc8bfcc0a1483b7-df57dee35f924801a23bd9b5c547a55f on tablet df57dee35f924801a23bd9b5c547a55f with peer e6ab42bc38694f7f9fc8bfcc0a1483b7
I20240501 14:09:38.960728 17850 ts_tablet_manager.cc:1888] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:38.960791 17850 log.cc:1196] T 40b29fb993f34e99ad3747a2ec7d6c4f P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/40b29fb993f34e99ad3747a2ec7d6c4f
I20240501 14:09:38.961364 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 40b29fb993f34e99ad3747a2ec7d6c4f (table rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]) successfully deleted
I20240501 14:09:38.966490 15090 raft_consensus.cc:1203] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:38.976200 17738 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:38.976677 17738 tablet_bootstrap.cc:492] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:38.976927 17738 ts_tablet_manager.cc:1367] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.007s
I20240501 14:09:38.977362 17738 raft_consensus.cc:348] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.977473 17738 raft_consensus.cc:729] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:38.977571 17738 consensus_queue.cc:260] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.977808 17738 ts_tablet_manager.cc:1398] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:38.978303 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-68b04992f1f94b36b01dfde24a42318d received from {username='slave'} at 127.6.82.5:37435
I20240501 14:09:38.978379 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-68b04992f1f94b36b01dfde24a42318d on tablet 68b04992f1f94b36b01dfde24a42318d with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:38.981753 17434 raft_consensus.cc:482] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:38.981865 17434 raft_consensus.cc:504] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.982270 17434 leader_election.cc:288] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:38.982539 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef85be81b074f65b7c022b03061064a" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:38.982652 14958 raft_consensus.cc:2454] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:38.982712 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef85be81b074f65b7c022b03061064a" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:38.982908 15090 raft_consensus.cc:2454] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:38.983067 15287 leader_election.cc:302] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.983246 17434 raft_consensus.cc:2781] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:38.983302 17434 raft_consensus.cc:482] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:38.983337 17434 raft_consensus.cc:3037] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.984401 17434 raft_consensus.cc:504] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.984778 17434 leader_election.cc:288] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:38.985286 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef85be81b074f65b7c022b03061064a" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:38.985414 14958 raft_consensus.cc:3037] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.986179 14958 raft_consensus.cc:2454] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:38.986433 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef85be81b074f65b7c022b03061064a" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:38.986464 15287 leader_election.cc:302] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:38.986558 15090 raft_consensus.cc:3037] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:38.986964 17434 raft_consensus.cc:2781] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:38.987066 17434 raft_consensus.cc:686] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:38.987191 17434 consensus_queue.cc:237] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:38.987562 15090 raft_consensus.cc:2454] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:38.989224 12442 catalog_manager.cc:5484] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:38.992444 12849 consensus_queue.cc:237] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:38.993767 12716 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:38.993845 12586 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:38.993973 12783 consensus_peers.cc:485] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01d9bd5e798f4bf2a5dd1eedcb4676cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:38.994273 17415 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.994907 17415 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:38.996431 15878 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.996853 12716 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.998980 12586 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:09:38.999655 12439 catalog_manager.cc:5484] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 4, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.003525 15353 raft_consensus.cc:1203] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:39.010474 17738 ts_tablet_manager.cc:902] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:39.011133 17738 tablet_copy_client.cc:308] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:39.011634 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet eb336cffd30b447b880af97da6882d27 from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:59219)
I20240501 14:09:39.011749 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet eb336cffd30b447b880af97da6882d27 from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:59219: session id = 8a13c028ba8a4a659b74af6c265df95e-eb336cffd30b447b880af97da6882d27
I20240501 14:09:39.012754 12607 tablet_copy_source_session.cc:205] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.013137 15353 consensus_queue.cc:237] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:39.014101 17738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336cffd30b447b880af97da6882d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.015324 15090 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.015667 17434 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:39.016250 17738 tablet_copy_client.cc:783] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.016404 17738 tablet_copy_client.cc:647] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.017184 12586 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:39.017380 15286 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 -> Peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): Couldn't send request to peer e6ab42bc38694f7f9fc8bfcc0a1483b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7130a0cb5bca433d8eb682e05b7a8f29. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:39.017690 17434 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:39.019065 17738 tablet_copy_client.cc:523] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.020226 17738 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:39.023213 17434 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:39.028023 12586 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:39.030905 15090 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:39.035197 12441 catalog_manager.cc:5484] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index -1 to 2, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:09:39.039986 17697 ts_tablet_manager.cc:902] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:39.041597 17697 tablet_copy_client.cc:308] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:09:39.042490 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 7693e3e204634676a758726bb6d3d2ce from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:34877)
I20240501 14:09:39.042606 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 7693e3e204634676a758726bb6d3d2ce from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:34877: session id = f23f426391724d6688aafffe7ee017d8-7693e3e204634676a758726bb6d3d2ce
I20240501 14:09:39.043705 12607 tablet_copy_source_session.cc:205] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.044543 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7693e3e204634676a758726bb6d3d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.049021 17697 tablet_copy_client.cc:783] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.049191 17697 tablet_copy_client.cc:647] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.049672 15090 consensus_queue.cc:237] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:39.051134 18070 ts_tablet_manager.cc:902] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: Initiating tablet copy from peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:39.051669 18070 tablet_copy_client.cc:308] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.6:37673
I20240501 14:09:39.051906 15222 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.052147 15110 tablet_copy_service.cc:140] P ab90e77f0027449bb3155980ca95593c: Received BeginTabletCopySession request for tablet d521005dcbf74e4fb9a67fe9992ce540 from peer a6e94562bee149c2ace25672f0f9bf50 ({username='slave'} at 127.6.82.4:51139)
I20240501 14:09:39.052246 15110 tablet_copy_service.cc:161] P ab90e77f0027449bb3155980ca95593c: Beginning new tablet copy session on tablet d521005dcbf74e4fb9a67fe9992ce540 from peer a6e94562bee149c2ace25672f0f9bf50 at {username='slave'} at 127.6.82.4:51139: session id = a6e94562bee149c2ace25672f0f9bf50-d521005dcbf74e4fb9a67fe9992ce540
W20240501 14:09:39.052274 15026 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d6a7ed55367493e9edd0461d6da6313. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:39.052649 15422 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:39.053181 15110 tablet_copy_source_session.cc:205] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.053226 15353 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.054085 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d521005dcbf74e4fb9a67fe9992ce540. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.054637 15422 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:39.055644 17738 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.055933 17738 tablet_bootstrap.cc:492] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:39.056519 17738 ts_tablet_manager.cc:1367] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.000s
I20240501 14:09:39.056993 17738 raft_consensus.cc:348] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:39.057118 17738 raft_consensus.cc:729] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:39.057266 18070 tablet_copy_client.cc:783] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.057224 17738 consensus_queue.cc:260] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:39.057531 18070 tablet_copy_client.cc:647] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.057612 17697 tablet_copy_client.cc:523] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.058681 17738 ts_tablet_manager.cc:1398] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.059018 17697 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:39.059585 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-eb336cffd30b447b880af97da6882d27 received from {username='slave'} at 127.6.82.5:59219
I20240501 14:09:39.059674 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-eb336cffd30b447b880af97da6882d27 on tablet eb336cffd30b447b880af97da6882d27 with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:39.060271 18070 tablet_copy_client.cc:523] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.062042 15422 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:39.064697 18070 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:39.064607 15222 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:39.067387 12442 catalog_manager.cc:5484] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.067417 15353 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:39.071425 18070 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.073509 18070 tablet_bootstrap.cc:492] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap complete.
I20240501 14:09:39.073702 18070 ts_tablet_manager.cc:1367] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:09:39.074910 18070 raft_consensus.cc:348] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:39.075035 18070 raft_consensus.cc:729] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: LEARNER
I20240501 14:09:39.075292 18070 consensus_queue.cc:260] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:39.075596 18070 ts_tablet_manager.cc:1398] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.077344 15110 tablet_copy_service.cc:342] P ab90e77f0027449bb3155980ca95593c: Request end of tablet copy session a6e94562bee149c2ace25672f0f9bf50-d521005dcbf74e4fb9a67fe9992ce540 received from {username='slave'} at 127.6.82.4:51139
I20240501 14:09:39.077435 15110 tablet_copy_service.cc:433] P ab90e77f0027449bb3155980ca95593c: ending tablet copy session a6e94562bee149c2ace25672f0f9bf50-d521005dcbf74e4fb9a67fe9992ce540 on tablet d521005dcbf74e4fb9a67fe9992ce540 with peer a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:39.087407 17751 ts_tablet_manager.cc:902] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:39.087841 17751 tablet_copy_client.cc:308] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:39.088395 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 188ce319682147ddaad30aae7232a311 from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:34873)
I20240501 14:09:39.088496 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 188ce319682147ddaad30aae7232a311 from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:34873: session id = 66e2dc34330848c3b8c2dd15898d34c8-188ce319682147ddaad30aae7232a311
I20240501 14:09:39.089474 12999 tablet_copy_source_session.cc:205] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.090474 17751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188ce319682147ddaad30aae7232a311. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.095315 15222 raft_consensus.cc:1203] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.097007 17751 tablet_copy_client.cc:783] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.097193 17751 tablet_copy_client.cc:647] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.098120 17697 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.098513 17697 tablet_bootstrap.cc:492] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:39.098820 17697 ts_tablet_manager.cc:1367] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.004s
I20240501 14:09:39.099406 17697 raft_consensus.cc:348] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:39.099620 17697 raft_consensus.cc:729] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:39.099800 17697 consensus_queue.cc:260] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:39.100831 17697 ts_tablet_manager.cc:1398] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.100821 17751 tablet_copy_client.cc:523] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.101413 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-7693e3e204634676a758726bb6d3d2ce received from {username='slave'} at 127.6.82.8:34877
I20240501 14:09:39.101511 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session f23f426391724d6688aafffe7ee017d8-7693e3e204634676a758726bb6d3d2ce on tablet 7693e3e204634676a758726bb6d3d2ce with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:39.102413 17751 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:39.102973 17925 raft_consensus.cc:482] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.103056 17925 raft_consensus.cc:504] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:39.103459 17925 leader_election.cc:288] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:39.105278 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b93a7b5dd24262a9e70e407653ed50" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:39.105465 12716 raft_consensus.cc:2379] T 26b93a7b5dd24262a9e70e407653ed50 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f23f426391724d6688aafffe7ee017d8 in current term 1: Already voted for candidate a6e94562bee149c2ace25672f0f9bf50 in this term.
I20240501 14:09:39.105877 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b93a7b5dd24262a9e70e407653ed50" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:39.106575 15288 leader_election.cc:302] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23f426391724d6688aafffe7ee017d8; no voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7
I20240501 14:09:39.106858 17925 raft_consensus.cc:3037] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.107842 17925 raft_consensus.cc:2726] T 26b93a7b5dd24262a9e70e407653ed50 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:39.110002 17751 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.110257 17751 tablet_bootstrap.cc:492] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:39.110450 17751 ts_tablet_manager.cc:1367] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:09:39.110971 17751 raft_consensus.cc:348] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:39.111081 17751 raft_consensus.cc:729] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:39.111191 17751 consensus_queue.cc:260] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:39.111451 17751 ts_tablet_manager.cc:1398] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:39.111959 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-188ce319682147ddaad30aae7232a311 received from {username='slave'} at 127.6.82.7:34873
I20240501 14:09:39.112037 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-188ce319682147ddaad30aae7232a311 on tablet 188ce319682147ddaad30aae7232a311 with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:39.114403 17167 consensus_queue.cc:1035] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.117623 16375 consensus_queue.cc:1035] T 26b93a7b5dd24262a9e70e407653ed50 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.121268 15353 raft_consensus.cc:1203] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.123715 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:39.127593 15090 raft_consensus.cc:1261] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.127648 14958 raft_consensus.cc:1261] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.128010 16475 consensus_queue.cc:1035] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.128152 17925 consensus_queue.cc:1035] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.130574 12439 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 8ef85be81b074f65b7c022b03061064a (table rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3]). Expected version 1 got 0
I20240501 14:09:39.131913 18116 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.131960 12442 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 8ef85be81b074f65b7c022b03061064a (table rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3]). Expected version 1 got 0
I20240501 14:09:39.132225 18116 tablet.cc:1620] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:39.133142 18117 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.133571 18117 tablet.cc:1620] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:39.135267 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3] alter complete (version 1)
I20240501 14:09:39.136140 12439 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 8ef85be81b074f65b7c022b03061064a (table rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3]). Expected version 1 got 0
I20240501 14:09:39.138806 18118 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.139954 15090 raft_consensus.cc:1203] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.140451 18118 tablet.cc:1620] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:39.174286 18070 ts_tablet_manager.cc:902] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:39.174830 18070 tablet_copy_client.cc:308] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:39.175326 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet ee9a1e9b13f546cc81aae6d729c8974f from peer a6e94562bee149c2ace25672f0f9bf50 ({username='slave'} at 127.6.82.4:40133)
I20240501 14:09:39.175439 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet ee9a1e9b13f546cc81aae6d729c8974f from peer a6e94562bee149c2ace25672f0f9bf50 at {username='slave'} at 127.6.82.4:40133: session id = a6e94562bee149c2ace25672f0f9bf50-ee9a1e9b13f546cc81aae6d729c8974f
I20240501 14:09:39.176463 12738 tablet_copy_source_session.cc:205] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.177348 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9a1e9b13f546cc81aae6d729c8974f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.179447 18070 tablet_copy_client.cc:783] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.179594 18070 tablet_copy_client.cc:647] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.181012 18070 tablet_copy_client.cc:523] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.182351 18070 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:39.185990 18070 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.186236 18070 tablet_bootstrap.cc:492] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: Bootstrap complete.
I20240501 14:09:39.186388 18070 ts_tablet_manager.cc:1367] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:09:39.186902 18070 raft_consensus.cc:348] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:39.187012 18070 raft_consensus.cc:729] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: LEARNER
I20240501 14:09:39.187114 18070 consensus_queue.cc:260] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:39.187405 18070 ts_tablet_manager.cc:1398] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:39.188081 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session a6e94562bee149c2ace25672f0f9bf50-ee9a1e9b13f546cc81aae6d729c8974f received from {username='slave'} at 127.6.82.4:40133
I20240501 14:09:39.188176 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session a6e94562bee149c2ace25672f0f9bf50-ee9a1e9b13f546cc81aae6d729c8974f on tablet ee9a1e9b13f546cc81aae6d729c8974f with peer a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:39.207532 18169 ts_tablet_manager.cc:902] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:39.208195 18169 tablet_copy_client.cc:308] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:39.208853 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet a0e8e476bc46420d992a43bf3b4c73fd from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58523)
I20240501 14:09:39.208971 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet a0e8e476bc46420d992a43bf3b4c73fd from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58523: session id = ab90e77f0027449bb3155980ca95593c-a0e8e476bc46420d992a43bf3b4c73fd
I20240501 14:09:39.209924 12738 tablet_copy_source_session.cc:205] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.210750 18169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e476bc46420d992a43bf3b4c73fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.213191 18169 tablet_copy_client.cc:783] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.213357 18169 tablet_copy_client.cc:647] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.214668 18169 tablet_copy_client.cc:523] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.216138 18169 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:39.239094 18169 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.239437 18169 tablet_bootstrap.cc:492] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:39.239385 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:39.239629 18169 ts_tablet_manager.cc:1367] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.000s
I20240501 14:09:39.240056 18169 raft_consensus.cc:348] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:39.240216 18169 raft_consensus.cc:729] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:39.240355 18169 consensus_queue.cc:260] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:39.240694 18169 ts_tablet_manager.cc:1398] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.241353 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-a0e8e476bc46420d992a43bf3b4c73fd received from {username='slave'} at 127.6.82.6:58523
I20240501 14:09:39.241434 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session ab90e77f0027449bb3155980ca95593c-a0e8e476bc46420d992a43bf3b4c73fd on tablet a0e8e476bc46420d992a43bf3b4c73fd with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:39.244812 18070 ts_tablet_manager.cc:902] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: Initiating tablet copy from peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:39.245263 18070 tablet_copy_client.cc:308] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.8:46801
I20240501 14:09:39.246598 15373 tablet_copy_service.cc:140] P f23f426391724d6688aafffe7ee017d8: Received BeginTabletCopySession request for tablet 565e6fb440dc4b00b3e2d0e1936889c8 from peer a6e94562bee149c2ace25672f0f9bf50 ({username='slave'} at 127.6.82.4:59519)
I20240501 14:09:39.246716 15373 tablet_copy_service.cc:161] P f23f426391724d6688aafffe7ee017d8: Beginning new tablet copy session on tablet 565e6fb440dc4b00b3e2d0e1936889c8 from peer a6e94562bee149c2ace25672f0f9bf50 at {username='slave'} at 127.6.82.4:59519: session id = a6e94562bee149c2ace25672f0f9bf50-565e6fb440dc4b00b3e2d0e1936889c8
I20240501 14:09:39.247678 15373 tablet_copy_source_session.cc:205] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.248612 18070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565e6fb440dc4b00b3e2d0e1936889c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.248944 18116 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:39.249256 18116 tablet.cc:1620] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:39.250886 18070 tablet_copy_client.cc:783] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.251044 18070 tablet_copy_client.cc:647] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.250999 18118 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:39.251307 18118 tablet.cc:1620] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:39.251963 18117 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:39.252252 18117 tablet.cc:1620] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:39.255734 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=39163954a23d4bf4871f4b7c66bce7e3] alter complete (version 2)
I20240501 14:09:39.257841 16475 consensus_queue.cc:1035] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.257964 18070 tablet_copy_client.cc:523] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.259564 18070 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:39.264092 18070 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.264312 18070 tablet_bootstrap.cc:492] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap complete.
I20240501 14:09:39.264467 18070 ts_tablet_manager.cc:1367] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:09:39.265038 18070 raft_consensus.cc:348] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:39.265175 18070 raft_consensus.cc:729] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: LEARNER
I20240501 14:09:39.265295 18070 consensus_queue.cc:260] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:09:39.265583 18070 ts_tablet_manager.cc:1398] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.266024 15373 tablet_copy_service.cc:342] P f23f426391724d6688aafffe7ee017d8: Request end of tablet copy session a6e94562bee149c2ace25672f0f9bf50-565e6fb440dc4b00b3e2d0e1936889c8 received from {username='slave'} at 127.6.82.4:59519
I20240501 14:09:39.266114 15373 tablet_copy_service.cc:433] P f23f426391724d6688aafffe7ee017d8: ending tablet copy session a6e94562bee149c2ace25672f0f9bf50-565e6fb440dc4b00b3e2d0e1936889c8 on tablet 565e6fb440dc4b00b3e2d0e1936889c8 with peer a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:39.269526 17925 consensus_queue.cc:1035] T a74a465ca78c43419b2066dcf077a691 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.277093 12442 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:39.284657 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3a66a73f304cd8b74db30afeeb59cb (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.284785 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3a66a73f304cd8b74db30afeeb59cb (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.284994 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3a66a73f304cd8b74db30afeeb59cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.285097 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3a66a73f304cd8b74db30afeeb59cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.285543 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 22e95701b67048ad871e7ff6fa4392cc (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.285827 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e95701b67048ad871e7ff6fa4392cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.285951 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 22e95701b67048ad871e7ff6fa4392cc (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.286224 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e95701b67048ad871e7ff6fa4392cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.287010 15069 tablet_service.cc:1467] Processing CreateTablet for tablet b744ec18c3474b11b72fbc2b160dbaef (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.287238 15069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b744ec18c3474b11b72fbc2b160dbaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.287310 17873 tablet_bootstrap.cc:492] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:39.288188 18210 tablet_bootstrap.cc:492] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:39.288220 17873 tablet_bootstrap.cc:654] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.288561 12827 tablet_service.cc:1467] Processing CreateTablet for tablet b744ec18c3474b11b72fbc2b160dbaef (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.288800 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b744ec18c3474b11b72fbc2b160dbaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.289340 17938 tablet_bootstrap.cc:492] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:39.289594 12566 tablet_service.cc:1467] Processing CreateTablet for tablet b744ec18c3474b11b72fbc2b160dbaef (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.289660 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 22e95701b67048ad871e7ff6fa4392cc (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.289856 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e95701b67048ad871e7ff6fa4392cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.289857 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b744ec18c3474b11b72fbc2b160dbaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.290632 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3a66a73f304cd8b74db30afeeb59cb (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=429458ada1064f9eadb89fb2c0a21537]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.290935 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d3a66a73f304cd8b74db30afeeb59cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.291642 18210 tablet_bootstrap.cc:654] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.292336 17938 tablet_bootstrap.cc:654] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.292902 18210 tablet_bootstrap.cc:492] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:39.292994 18210 ts_tablet_manager.cc:1367] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:39.293018 18209 tablet_bootstrap.cc:492] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:39.293462 18210 raft_consensus.cc:348] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.293582 18210 raft_consensus.cc:374] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.293615 18210 raft_consensus.cc:729] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.293731 18210 consensus_queue.cc:260] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.293959 18209 tablet_bootstrap.cc:654] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.294013 18210 ts_tablet_manager.cc:1398] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.294656 18213 tablet_bootstrap.cc:492] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:39.294796 18215 tablet_bootstrap.cc:492] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:39.295610 18213 tablet_bootstrap.cc:654] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.296147 18215 tablet_bootstrap.cc:654] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.296548 17938 tablet_bootstrap.cc:492] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:39.296643 17938 ts_tablet_manager.cc:1367] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:09:39.297158 17938 raft_consensus.cc:348] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.297245 17938 raft_consensus.cc:374] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.297264 17938 raft_consensus.cc:729] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.297309 18215 tablet_bootstrap.cc:492] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:39.297387 18215 ts_tablet_manager.cc:1367] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:39.297369 17938 consensus_queue.cc:260] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.297637 17938 ts_tablet_manager.cc:1398] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.297791 17938 tablet_bootstrap.cc:492] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:39.297796 18215 raft_consensus.cc:348] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.297921 18215 raft_consensus.cc:374] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.297952 18215 raft_consensus.cc:729] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.298070 18215 consensus_queue.cc:260] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.298348 18215 ts_tablet_manager.cc:1398] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.298760 17938 tablet_bootstrap.cc:654] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.299708 17873 tablet_bootstrap.cc:492] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:39.299800 17873 ts_tablet_manager.cc:1367] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20240501 14:09:39.300274 17873 raft_consensus.cc:348] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.300376 17873 raft_consensus.cc:374] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.300421 17873 raft_consensus.cc:729] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.300568 17873 consensus_queue.cc:260] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.300900 17873 ts_tablet_manager.cc:1398] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:39.301081 17873 tablet_bootstrap.cc:492] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:39.301982 17873 tablet_bootstrap.cc:654] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.302887 17697 ts_tablet_manager.cc:902] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:39.303368 17697 tablet_copy_client.cc:308] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:39.303689 17873 tablet_bootstrap.cc:492] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:39.303822 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet 6628290de33e4aba8ccd1651d1d054dd from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:35333)
I20240501 14:09:39.303922 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet 6628290de33e4aba8ccd1651d1d054dd from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:35333: session id = f23f426391724d6688aafffe7ee017d8-6628290de33e4aba8ccd1651d1d054dd
I20240501 14:09:39.304224 17938 tablet_bootstrap.cc:492] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:39.304296 17938 ts_tablet_manager.cc:1367] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:39.304790 17938 raft_consensus.cc:348] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.304888 17938 raft_consensus.cc:374] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.304919 17938 raft_consensus.cc:729] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.304947 12738 tablet_copy_source_session.cc:205] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.305060 17938 consensus_queue.cc:260] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.305492 17938 ts_tablet_manager.cc:1398] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.306706 17873 ts_tablet_manager.cc:1367] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:09:39.306943 18209 tablet_bootstrap.cc:492] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:39.307027 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6628290de33e4aba8ccd1651d1d054dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.307044 18209 ts_tablet_manager.cc:1367] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:39.307250 17873 raft_consensus.cc:348] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.307370 17873 raft_consensus.cc:374] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.307404 17873 raft_consensus.cc:729] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.307461 18209 raft_consensus.cc:348] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.307567 18209 raft_consensus.cc:374] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.307626 18209 raft_consensus.cc:729] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.307760 18209 consensus_queue.cc:260] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.308104 18209 ts_tablet_manager.cc:1398] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.308317 17873 consensus_queue.cc:260] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.308616 17873 ts_tablet_manager.cc:1398] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.308750 18209 tablet_bootstrap.cc:492] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:39.309187 17697 tablet_copy_client.cc:783] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.309326 17697 tablet_copy_client.cc:647] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.310163 18213 tablet_bootstrap.cc:492] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:39.310346 18213 ts_tablet_manager.cc:1367] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:39.310720 17697 tablet_copy_client.cc:523] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.310922 18213 raft_consensus.cc:348] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.311019 18213 raft_consensus.cc:374] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.311046 18213 raft_consensus.cc:729] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.311172 18213 consensus_queue.cc:260] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.311477 18213 ts_tablet_manager.cc:1398] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:39.312079 17697 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:39.313246 18216 consensus_queue.cc:1035] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.314002 18209 tablet_bootstrap.cc:654] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.315325 18209 tablet_bootstrap.cc:492] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:39.315392 18209 ts_tablet_manager.cc:1367] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:39.315886 18209 raft_consensus.cc:348] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.316088 18209 raft_consensus.cc:374] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.316170 18209 raft_consensus.cc:729] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.316349 18209 consensus_queue.cc:260] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.317026 18209 ts_tablet_manager.cc:1398] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:39.317392 17160 consensus_queue.cc:1035] T db946894cd2348b0be6b8bb574c19667 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.335489 17177 raft_consensus.cc:482] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.335613 17177 raft_consensus.cc:504] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.336035 17177 leader_election.cc:288] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:39.336561 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3a66a73f304cd8b74db30afeeb59cb" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:39.336699 15353 raft_consensus.cc:2454] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:39.337011 12654 leader_election.cc:302] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:39.337194 17177 raft_consensus.cc:2781] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.337265 17177 raft_consensus.cc:482] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.337317 17177 raft_consensus.cc:3037] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.337574 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3a66a73f304cd8b74db30afeeb59cb" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:39.337710 12849 raft_consensus.cc:2454] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:39.337869 17697 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.338160 17697 tablet_bootstrap.cc:492] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:39.338305 17177 raft_consensus.cc:504] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.338333 17697 ts_tablet_manager.cc:1367] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.003s
I20240501 14:09:39.338629 17177 leader_election.cc:288] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:39.338910 17697 raft_consensus.cc:348] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:39.339023 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3a66a73f304cd8b74db30afeeb59cb" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:39.339030 17697 raft_consensus.cc:729] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:39.339152 15353 raft_consensus.cc:3037] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.339156 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3a66a73f304cd8b74db30afeeb59cb" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:39.339255 12849 raft_consensus.cc:3037] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.339260 17697 consensus_queue.cc:260] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:39.339571 17697 ts_tablet_manager.cc:1398] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 14:09:39.340032 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-6628290de33e4aba8ccd1651d1d054dd received from {username='slave'} at 127.6.82.8:35333
I20240501 14:09:39.340059 15353 raft_consensus.cc:2454] T 9d3a66a73f304cd8b74db30afeeb59cb P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:39.340098 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session f23f426391724d6688aafffe7ee017d8-6628290de33e4aba8ccd1651d1d054dd on tablet 6628290de33e4aba8ccd1651d1d054dd with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:39.340278 12849 raft_consensus.cc:2454] T 9d3a66a73f304cd8b74db30afeeb59cb P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:39.340330 12654 leader_election.cc:302] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:39.340476 17177 raft_consensus.cc:2781] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.340566 17177 raft_consensus.cc:686] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:39.340713 17177 consensus_queue.cc:237] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:39.342670 12439 catalog_manager.cc:5484] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.357293 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:39.362967 18116 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.364800 18117 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.367283 18118 tablet.cc:1722] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:39.371197 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=39163954a23d4bf4871f4b7c66bce7e3] alter complete (version 3)
I20240501 14:09:39.377080 16475 raft_consensus.cc:482] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.377195 16475 raft_consensus.cc:504] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.377631 16475 leader_election.cc:288] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:39.378055 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e95701b67048ad871e7ff6fa4392cc" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:39.378208 15222 raft_consensus.cc:2454] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:39.378492 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e95701b67048ad871e7ff6fa4392cc" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:39.378540 15286 leader_election.cc:302] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:39.378662 15089 raft_consensus.cc:2454] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:39.378871 16475 raft_consensus.cc:2781] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.378938 16475 raft_consensus.cc:482] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.378983 16475 raft_consensus.cc:3037] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.380004 16475 raft_consensus.cc:504] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.380354 16475 leader_election.cc:288] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:39.380832 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e95701b67048ad871e7ff6fa4392cc" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:39.380944 15090 raft_consensus.cc:3037] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.381615 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e95701b67048ad871e7ff6fa4392cc" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:39.381736 15222 raft_consensus.cc:3037] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.381965 15090 raft_consensus.cc:2454] T 22e95701b67048ad871e7ff6fa4392cc P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:39.382262 15288 leader_election.cc:302] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:39.382467 16475 raft_consensus.cc:2781] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.382577 16475 raft_consensus.cc:686] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:39.382692 15222 raft_consensus.cc:2454] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:39.382719 16475 consensus_queue.cc:237] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:39.384864 12439 catalog_manager.cc:5484] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.391270 12718 raft_consensus.cc:1203] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:39.418517 18010 ts_tablet_manager.cc:902] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Initiating tablet copy from peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:39.419157 18010 tablet_copy_client.cc:308] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.8:46801
I20240501 14:09:39.419705 15373 tablet_copy_service.cc:140] P f23f426391724d6688aafffe7ee017d8: Received BeginTabletCopySession request for tablet 7130a0cb5bca433d8eb682e05b7a8f29 from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 ({username='slave'} at 127.6.82.2:36499)
I20240501 14:09:39.419828 15373 tablet_copy_service.cc:161] P f23f426391724d6688aafffe7ee017d8: Beginning new tablet copy session on tablet 7130a0cb5bca433d8eb682e05b7a8f29 from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 at {username='slave'} at 127.6.82.2:36499: session id = e6ab42bc38694f7f9fc8bfcc0a1483b7-7130a0cb5bca433d8eb682e05b7a8f29
I20240501 14:09:39.420892 15373 tablet_copy_source_session.cc:205] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.422508 18010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7130a0cb5bca433d8eb682e05b7a8f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.424567 18010 tablet_copy_client.cc:783] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.424765 18010 tablet_copy_client.cc:647] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.425087 15655 raft_consensus.cc:1050] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:39.425529 15655 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:39.425971 18010 tablet_copy_client.cc:523] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.427151 15353 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:39.427332 18010 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:39.427590 14958 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:39.427609 18238 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:39.428113 12978 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:39.428901 18238 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:39.429083 17272 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:39.431123 18239 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.431351 18010 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.431586 18010 tablet_bootstrap.cc:492] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap complete.
I20240501 14:09:39.431483 14958 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.431483 12978 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.431733 18010 ts_tablet_manager.cc:1367] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 14:09:39.432178 18010 raft_consensus.cc:348] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:39.432310 18010 raft_consensus.cc:729] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: LEARNER
I20240501 14:09:39.432446 18010 consensus_queue.cc:260] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:09:39.433061 18010 ts_tablet_manager.cc:1398] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 14:09:39.434384 12441 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.435693 15353 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.435981 15373 tablet_copy_service.cc:342] P f23f426391724d6688aafffe7ee017d8: Request end of tablet copy session e6ab42bc38694f7f9fc8bfcc0a1483b7-7130a0cb5bca433d8eb682e05b7a8f29 received from {username='slave'} at 127.6.82.2:36499
I20240501 14:09:39.436054 15373 tablet_copy_service.cc:433] P f23f426391724d6688aafffe7ee017d8: ending tablet copy session e6ab42bc38694f7f9fc8bfcc0a1483b7-7130a0cb5bca433d8eb682e05b7a8f29 on tablet 7130a0cb5bca433d8eb682e05b7a8f29 with peer e6ab42bc38694f7f9fc8bfcc0a1483b7
I20240501 14:09:39.441596 12586 consensus_queue.cc:237] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:39.442806 15353 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:39.442867 14958 raft_consensus.cc:1261] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:39.443224 17272 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:39.443449 18239 consensus_queue.cc:1035] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:39.445080 18239 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.445631 14958 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.445724 15353 raft_consensus.cc:2932] T d178c3291efc44369e3adb921ccc6517 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.447494 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d178c3291efc44369e3adb921ccc6517 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:39.449172 12442 catalog_manager.cc:5484] T d178c3291efc44369e3adb921ccc6517 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 8 to 9, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.449421 17415 raft_consensus.cc:482] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.449532 17415 raft_consensus.cc:504] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.449939 17415 leader_election.cc:288] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:39.450265 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b744ec18c3474b11b72fbc2b160dbaef" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:39.450326 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b744ec18c3474b11b72fbc2b160dbaef" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:39.450407 15090 raft_consensus.cc:2454] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:39.450455 12586 raft_consensus.cc:2454] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:39.450794 12784 leader_election.cc:302] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:39.450961 17415 raft_consensus.cc:2781] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.451117 17415 raft_consensus.cc:482] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.451174 17415 raft_consensus.cc:3037] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.452257 17415 raft_consensus.cc:504] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.452483 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet d178c3291efc44369e3adb921ccc6517 with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:39.452608 17415 leader_election.cc:288] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:39.452822 18244 tablet_replica.cc:330] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:39.453007 18244 raft_consensus.cc:2229] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.453019 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b744ec18c3474b11b72fbc2b160dbaef" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:39.453132 15090 raft_consensus.cc:3037] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.453284 18244 raft_consensus.cc:2258] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.453824 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b744ec18c3474b11b72fbc2b160dbaef" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:39.453929 12586 raft_consensus.cc:3037] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.454130 15090 raft_consensus.cc:2454] T b744ec18c3474b11b72fbc2b160dbaef P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:39.454015 18244 ts_tablet_manager.cc:1875] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:39.454562 12784 leader_election.cc:302] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:39.454818 17415 raft_consensus.cc:2781] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.454895 17415 raft_consensus.cc:686] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:39.454931 12586 raft_consensus.cc:2454] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:39.455017 17415 consensus_queue.cc:237] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:39.456876 18244 ts_tablet_manager.cc:1888] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:39.456959 18244 log.cc:1196] T d178c3291efc44369e3adb921ccc6517 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/d178c3291efc44369e3adb921ccc6517
I20240501 14:09:39.457496 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet d178c3291efc44369e3adb921ccc6517 (table rebalance_test_table_15 [id=ae91e0302c1c4c80a6ae85bad9bf91b4]) successfully deleted
I20240501 14:09:39.457731 12441 catalog_manager.cc:5484] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.473943 12441 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:39.474109 12441 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:39.476284 12441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 8ef85be81b074f65b7c022b03061064a
I20240501 14:09:39.477174 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef85be81b074f65b7c022b03061064a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:39.477277 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef85be81b074f65b7c022b03061064a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:39.477397 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef85be81b074f65b7c022b03061064a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:39 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:39.477680 18258 tablet_replica.cc:330] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:39.477833 18258 raft_consensus.cc:2229] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.477958 18258 raft_consensus.cc:2258] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.478369 18257 tablet_replica.cc:330] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:39.478507 18257 raft_consensus.cc:2229] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.478646 18257 raft_consensus.cc:2258] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.478946 18256 tablet_replica.cc:330] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:39.478766 18258 ts_tablet_manager.cc:1875] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.479087 18256 raft_consensus.cc:2229] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:39.479252 18256 raft_consensus.cc:2258] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.479677 18257 ts_tablet_manager.cc:1875] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.481376 18258 ts_tablet_manager.cc:1888] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.481539 18258 log.cc:1196] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/8ef85be81b074f65b7c022b03061064a
I20240501 14:09:39.481804 18257 ts_tablet_manager.cc:1888] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.481916 18258 ts_tablet_manager.cc:1909] T 8ef85be81b074f65b7c022b03061064a P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:39.482371 18256 ts_tablet_manager.cc:1875] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:39.482517 18257 log.cc:1196] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/8ef85be81b074f65b7c022b03061064a
I20240501 14:09:39.482929 18257 ts_tablet_manager.cc:1909] T 8ef85be81b074f65b7c022b03061064a P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:39.482923 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 8ef85be81b074f65b7c022b03061064a (table rebalancer_dynamic_table_new_name [id=39163954a23d4bf4871f4b7c66bce7e3]) successfully deleted
I20240501 14:09:39.484210 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 8ef85be81b074f65b7c022b03061064a (table rebalancer_dynamic_table_new_name [id=39163954a23d4bf4871f4b7c66bce7e3]) successfully deleted
I20240501 14:09:39.485479 18256 ts_tablet_manager.cc:1888] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:39.485626 18256 log.cc:1196] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/8ef85be81b074f65b7c022b03061064a
I20240501 14:09:39.485688 15353 raft_consensus.cc:1203] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:39.486028 18256 ts_tablet_manager.cc:1909] T 8ef85be81b074f65b7c022b03061064a P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:39.488626 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 8ef85be81b074f65b7c022b03061064a (table rebalancer_dynamic_table_new_name [id=39163954a23d4bf4871f4b7c66bce7e3]) successfully deleted
I20240501 14:09:39.513017 16837 raft_consensus.cc:1050] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:39.513500 16837 consensus_queue.cc:237] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:39.515425 12978 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.515425 12849 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.515429 14958 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.515846 16837 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.516018 18286 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.516839 16837 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.518368 18286 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.518846 12849 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.520411 12978 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.521495 12441 catalog_manager.cc:5484] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.522354 14958 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.528156 14958 raft_consensus.cc:1203] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:39.529174 12718 consensus_queue.cc:237] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:39.530248 14958 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.530611 12978 raft_consensus.cc:1261] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.530853 18286 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.531627 18286 consensus_queue.cc:1035] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.532740 18287 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.533190 14958 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.533617 12978 raft_consensus.cc:2932] T 16324878552c4b6dbf10e8d5f73a0436 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:39.534799 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16324878552c4b6dbf10e8d5f73a0436 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:39.536885 12439 catalog_manager.cc:5484] T 16324878552c4b6dbf10e8d5f73a0436 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.539649 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 16324878552c4b6dbf10e8d5f73a0436 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:39.540040 18301 tablet_replica.cc:330] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:39.540416 18301 raft_consensus.cc:2229] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.540663 18301 raft_consensus.cc:2258] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.541342 18301 ts_tablet_manager.cc:1875] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:39.544099 18301 ts_tablet_manager.cc:1888] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:39.544188 18301 log.cc:1196] T 16324878552c4b6dbf10e8d5f73a0436 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/16324878552c4b6dbf10e8d5f73a0436
I20240501 14:09:39.545073 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 16324878552c4b6dbf10e8d5f73a0436 (table rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]) successfully deleted
I20240501 14:09:39.548013 18238 raft_consensus.cc:1050] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:39.548480 18238 consensus_queue.cc:237] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:39.550199 12718 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.550241 15222 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.550503 12849 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.550925 17272 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.551200 15655 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.551563 15655 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.553568 15655 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:39.554376 12718 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:39.554543 15222 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:39.556591 12849 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:39.558271 12439 catalog_manager.cc:5484] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.562875 17415 raft_consensus.cc:1050] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:39.563283 17415 consensus_queue.cc:237] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:39.564704 15090 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.564703 12718 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.564865 12978 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.565120 17415 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.565287 16502 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.565415 15878 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.568063 18321 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.568490 12978 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.569726 15090 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.572234 12443 catalog_manager.cc:5484] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.572938 12718 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.573539 12978 raft_consensus.cc:1203] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.579265 12849 consensus_queue.cc:237] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:39.579973 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:39.581036 15090 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.581056 12718 raft_consensus.cc:1261] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.581446 17415 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.582788 17415 consensus_queue.cc:1035] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.583978 17415 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.584858 15090 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.586344 14937 tablet_service.cc:1467] Processing CreateTablet for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]), partition=
I20240501 14:09:39.586577 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf3fa8e0dec4ceb9057f70480aa5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.586916 12566 tablet_service.cc:1467] Processing CreateTablet for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]), partition=
I20240501 14:09:39.587080 12827 tablet_service.cc:1467] Processing CreateTablet for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]), partition=
I20240501 14:09:39.587157 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf3fa8e0dec4ceb9057f70480aa5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.587268 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf3fa8e0dec4ceb9057f70480aa5318. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.589124 18213 tablet_bootstrap.cc:492] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:39.589370 12439 catalog_manager.cc:5484] T 727a51852ac2428eb23f7690fec56a16 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.590097 18213 tablet_bootstrap.cc:654] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.590260 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 727a51852ac2428eb23f7690fec56a16 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:39.590898 18331 tablet_bootstrap.cc:492] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:39.591255 18213 tablet_bootstrap.cc:492] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:39.591339 18213 ts_tablet_manager.cc:1367] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:39.591439 18209 tablet_bootstrap.cc:492] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:39.591778 18213 raft_consensus.cc:348] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.591904 18213 raft_consensus.cc:374] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.591938 18213 raft_consensus.cc:729] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.591938 18331 tablet_bootstrap.cc:654] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.592067 18213 consensus_queue.cc:260] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.592271 18209 tablet_bootstrap.cc:654] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.592393 18213 ts_tablet_manager.cc:1398] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.593314 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 727a51852ac2428eb23f7690fec56a16 with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:39.593578 18244 tablet_replica.cc:330] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:39.593746 18244 raft_consensus.cc:2229] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.593902 18244 raft_consensus.cc:2258] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.596279 18209 tablet_bootstrap.cc:492] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:39.596388 18209 ts_tablet_manager.cc:1367] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:39.596838 18209 raft_consensus.cc:348] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.596951 18209 raft_consensus.cc:374] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.596992 18209 raft_consensus.cc:729] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.597100 18209 consensus_queue.cc:260] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.597398 18209 ts_tablet_manager.cc:1398] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:39.597775 18335 ts_tablet_manager.cc:902] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:39.598016 12718 raft_consensus.cc:2932] T 727a51852ac2428eb23f7690fec56a16 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.598315 18335 tablet_copy_client.cc:308] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:39.598820 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc from peer 8a13c028ba8a4a659b74af6c265df95e ({username='slave'} at 127.6.82.5:42185)
I20240501 14:09:39.598899 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc from peer 8a13c028ba8a4a659b74af6c265df95e at {username='slave'} at 127.6.82.5:42185: session id = 8a13c028ba8a4a659b74af6c265df95e-01d9bd5e798f4bf2a5dd1eedcb4676cc
I20240501 14:09:39.599913 12869 tablet_copy_source_session.cc:205] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.601877 18244 ts_tablet_manager.cc:1875] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:39.602715 18335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.604033 18331 tablet_bootstrap.cc:492] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:39.604130 18331 ts_tablet_manager.cc:1367] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:39.604307 18244 ts_tablet_manager.cc:1888] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:39.604389 18244 log.cc:1196] T 727a51852ac2428eb23f7690fec56a16 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/727a51852ac2428eb23f7690fec56a16
I20240501 14:09:39.605037 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 727a51852ac2428eb23f7690fec56a16 (table rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]) successfully deleted
I20240501 14:09:39.605016 18331 raft_consensus.cc:348] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.606387 18331 raft_consensus.cc:374] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:39.606436 18331 raft_consensus.cc:729] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:39.606592 18331 consensus_queue.cc:260] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.607054 18331 ts_tablet_manager.cc:1398] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:39.610639 18335 tablet_copy_client.cc:783] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.610826 18335 tablet_copy_client.cc:647] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.612006 18335 tablet_copy_client.cc:523] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.613430 18335 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:39.620467 15090 raft_consensus.cc:1203] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.629554 14958 raft_consensus.cc:1203] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.638176 15222 raft_consensus.cc:1203] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:09:39.641268 18335 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.641570 18335 tablet_bootstrap.cc:492] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap complete.
I20240501 14:09:39.641772 18335 ts_tablet_manager.cc:1367] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.004s
I20240501 14:09:39.642278 18335 raft_consensus.cc:348] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:39.642410 18335 raft_consensus.cc:729] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: LEARNER
I20240501 14:09:39.642504 18335 consensus_queue.cc:260] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
I20240501 14:09:39.642993 18335 ts_tablet_manager.cc:1398] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240501 14:09:39.643807 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session 8a13c028ba8a4a659b74af6c265df95e-01d9bd5e798f4bf2a5dd1eedcb4676cc received from {username='slave'} at 127.6.82.5:42185
I20240501 14:09:39.643898 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session 8a13c028ba8a4a659b74af6c265df95e-01d9bd5e798f4bf2a5dd1eedcb4676cc on tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc with peer 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:09:39.644052 12978 raft_consensus.cc:1203] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.653216 18361 ts_tablet_manager.cc:902] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:39.653817 18361 tablet_copy_client.cc:308] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.6:37673
I20240501 14:09:39.654911 15110 tablet_copy_service.cc:140] P ab90e77f0027449bb3155980ca95593c: Received BeginTabletCopySession request for tablet 3d6a7ed55367493e9edd0461d6da6313 from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:59095)
I20240501 14:09:39.655033 15110 tablet_copy_service.cc:161] P ab90e77f0027449bb3155980ca95593c: Beginning new tablet copy session on tablet 3d6a7ed55367493e9edd0461d6da6313 from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:59095: session id = 4902d2609e6d44d089590a42e70356b2-3d6a7ed55367493e9edd0461d6da6313
I20240501 14:09:39.655997 15110 tablet_copy_source_session.cc:205] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:39.656060 12586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6"
dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34654
I20240501 14:09:39.656260 12586 raft_consensus.cc:595] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:39.657099 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6a7ed55367493e9edd0461d6da6313. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.659710 18361 tablet_copy_client.cc:783] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:39.659929 18361 tablet_copy_client.cc:647] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:39.661033 18361 tablet_copy_client.cc:523] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:39.662631 12978 raft_consensus.cc:1203] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.663784 18361 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:39.668226 18361 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:39.668545 18361 tablet_bootstrap.cc:492] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:09:39.668751 18361 ts_tablet_manager.cc:1367] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240501 14:09:39.669152 18361 raft_consensus.cc:348] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:39.669258 18361 raft_consensus.cc:729] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:09:39.669384 18361 consensus_queue.cc:260] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:39.669643 18361 ts_tablet_manager.cc:1398] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:39.670181 15110 tablet_copy_service.cc:342] P ab90e77f0027449bb3155980ca95593c: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-3d6a7ed55367493e9edd0461d6da6313 received from {username='slave'} at 127.6.82.3:59095
I20240501 14:09:39.670276 15110 tablet_copy_service.cc:433] P ab90e77f0027449bb3155980ca95593c: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-3d6a7ed55367493e9edd0461d6da6313 on tablet 3d6a7ed55367493e9edd0461d6da6313 with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:09:39.670460 18321 raft_consensus.cc:1050] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:39.670892 18321 consensus_queue.cc:237] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:39.672614 12586 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.672639 12978 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.673163 17415 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.673297 15353 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.673339 15878 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.674014 17415 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.676772 15878 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.677210 15353 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.677230 12586 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.679760 12439 catalog_manager.cc:5484] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.681226 12978 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.685680 12849 consensus_queue.cc:237] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:39.687129 15353 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.687374 12978 raft_consensus.cc:1261] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.687532 17415 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.687763 18321 consensus_queue.cc:1035] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.689445 17415 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.689867 15353 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.690192 12978 raft_consensus.cc:2932] T 5648b7e0a1e744cea470feb7911d3d2c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.692284 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5648b7e0a1e744cea470feb7911d3d2c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:39.692955 16502 raft_consensus.cc:1050] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:39.693351 16502 consensus_queue.cc:237] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:39.694202 12443 catalog_manager.cc:5484] T 5648b7e0a1e744cea470feb7911d3d2c P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 5 to 6, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.695308 12718 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.695708 16502 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.696317 15090 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.696303 12978 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:39.698096 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 5648b7e0a1e744cea470feb7911d3d2c with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:39.698459 18394 tablet_replica.cc:330] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:39.698468 15222 consensus_queue.cc:237] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:39.698648 18394 raft_consensus.cc:2229] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:39.698810 18321 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.698884 18394 raft_consensus.cc:2258] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:39.698992 16502 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:39.699709 18394 ts_tablet_manager.cc:1875] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:39.700775 12718 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.701228 18216 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:39.701450 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8226993467944dcb24d1efcda07538a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:39.702087 12979 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.702540 18216 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:39.703341 18321 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.705459 18394 ts_tablet_manager.cc:1888] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:39.705591 18394 log.cc:1196] T 5648b7e0a1e744cea470feb7911d3d2c P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/5648b7e0a1e744cea470feb7911d3d2c
I20240501 14:09:39.706218 15090 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.706662 12716 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.706949 12979 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.709110 12442 catalog_manager.cc:5484] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:39.710073 18216 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:39.710249 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 5648b7e0a1e744cea470feb7911d3d2c (table rebalance_test_table_7 [id=464445a14fcc47deaab4fe199f4f101c]) successfully deleted
I20240501 14:09:39.712548 12979 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:39.713248 12718 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:39.714632 12441 catalog_manager.cc:5484] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index -1 to 2, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.724206 12979 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:39.725589 12849 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:39.725589 14958 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:39.726008 16375 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:39.726186 18279 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:09:39.726649 12916 consensus_peers.cc:485] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63a76e89d58d48ba9d1b3b98ef6c172d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:39.728175 15353 raft_consensus.cc:1203] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:39.728430 18279 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:39.728813 12849 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:39.731441 14958 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:39.731758 12440 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 6 to 7, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:09:39.769508 18287 consensus_queue.cc:1035] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.774012 17177 consensus_queue.cc:1035] T 9d3a66a73f304cd8b74db30afeeb59cb P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.813668 18239 raft_consensus.cc:482] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:39.813782 18239 raft_consensus.cc:504] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.814200 18239 leader_election.cc:288] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:39.814570 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf3fa8e0dec4ceb9057f70480aa5318" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:39.814594 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf3fa8e0dec4ceb9057f70480aa5318" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:39.814711 14958 raft_consensus.cc:2454] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:39.814721 12849 raft_consensus.cc:2454] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:39.815053 12523 leader_election.cc:302] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:39.815215 18239 raft_consensus.cc:2781] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:39.815263 18239 raft_consensus.cc:482] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:39.815310 18239 raft_consensus.cc:3037] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.816130 18239 raft_consensus.cc:504] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.816454 18239 leader_election.cc:288] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:39.816787 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf3fa8e0dec4ceb9057f70480aa5318" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:39.816856 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf3fa8e0dec4ceb9057f70480aa5318" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:39.816905 14958 raft_consensus.cc:3037] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.816970 12849 raft_consensus.cc:3037] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:39.817672 14958 raft_consensus.cc:2454] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:39.817948 12849 raft_consensus.cc:2454] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:39.818207 12521 leader_election.cc:302] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:39.818449 18239 raft_consensus.cc:2781] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:39.818540 18239 raft_consensus.cc:686] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:39.818701 18239 consensus_queue.cc:237] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:39.821308 12441 catalog_manager.cc:5484] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.842334 12586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6"
dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34698
I20240501 14:09:39.842499 12586 raft_consensus.cc:595] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:39.862691 12849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5957da39df64253a2b8d7f388308719"
dest_uuid: "4902d2609e6d44d089590a42e70356b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41404
I20240501 14:09:39.862937 12849 raft_consensus.cc:595] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:39.886835 12716 raft_consensus.cc:1203] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:39.907442 16523 raft_consensus.cc:1050] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: attempting to promote NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 to VOTER
I20240501 14:09:39.907900 16523 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:39.909250 12716 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.909358 12848 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.909842 18544 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.909991 15090 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:39.911226 16523 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.911415 18541 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:39.915251 18544 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:39.916589 12716 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:39.918435 12848 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:39.918570 18520 consensus_queue.cc:1035] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.918905 15090 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:39.920930 12442 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 5 to 6, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.925910 17925 consensus_queue.cc:1035] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.940270 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:39.945008 12848 raft_consensus.cc:1261] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.945194 14958 raft_consensus.cc:1261] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:39.945459 15655 consensus_queue.cc:1035] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.945680 18239 consensus_queue.cc:1035] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:39.947944 18239 raft_consensus.cc:1050] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:39.948372 18239 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:39.948577 12440 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]). Expected version 1 got 0
I20240501 14:09:39.949388 12441 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]). Expected version 1 got 0
I20240501 14:09:39.950105 15222 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:39.950852 18238 raft_consensus.cc:979] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: : Instructing follower 4902d2609e6d44d089590a42e70356b2 to start an election
I20240501 14:09:39.950956 18238 raft_consensus.cc:1067] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Signalling peer 4902d2609e6d44d089590a42e70356b2 to start an election
I20240501 14:09:39.951099 17272 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:39.952013 15353 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:39.952941 12848 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:39.953212 18239 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:39.953804 18578 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.954207 18578 tablet.cc:1620] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:39.954226 18579 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.954320 12848 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6"
dest_uuid: "4902d2609e6d44d089590a42e70356b2"
from {username='slave'} at 127.6.82.1:38293
I20240501 14:09:39.954463 12848 raft_consensus.cc:482] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:39.954532 12848 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:39.954900 18579 tablet.cc:1620] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:39.956398 12848 raft_consensus.cc:504] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:39.956719 12441 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]). Expected version 1 got 0
I20240501 14:09:39.956863 12848 leader_election.cc:288] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:39.957491 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:39.957600 12586 raft_consensus.cc:3032] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:39.957639 12586 raft_consensus.cc:729] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:39.957798 12586 consensus_queue.cc:260] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:39.957970 12586 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:39.957940 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:39.958060 12716 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:39.959126 12586 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 2.
I20240501 14:09:39.959182 12716 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 2.
I20240501 14:09:39.959237 18580 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:39.959615 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1a9d0e6943dc880161ae320673b6" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:39.959749 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4] alter complete (version 1)
I20240501 14:09:39.959772 15222 raft_consensus.cc:3037] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:39.960134 18580 tablet.cc:1620] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:39.960824 15222 raft_consensus.cc:2454] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 2.
I20240501 14:09:39.962666 18238 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:39.963042 12441 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]). Expected version 1 got 0
I20240501 14:09:39.963457 12782 leader_election.cc:302] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:39.963645 17415 raft_consensus.cc:2781] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:39.963747 17415 raft_consensus.cc:686] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 2 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:39.963896 17415 consensus_queue.cc:237] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:39.966989 12440 catalog_manager.cc:5484] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 1 to 2, leader changed from 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 2 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:39.967599 15655 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.968086 12848 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.969801 15353 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.970028 15222 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:39.970589 12441 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:39.987864 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:39.995193 14938 tablet_service.cc:1467] Processing CreateTablet for tablet cfbc588ca41e46b080296610f6739201 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.995519 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfbc588ca41e46b080296610f6739201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.995793 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 052117f3357f4577b80c0812b3ef2b05 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.996034 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052117f3357f4577b80c0812b3ef2b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.997805 15202 tablet_service.cc:1467] Processing CreateTablet for tablet c44df3acaa1a4f0d90f6553449d296a8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.997874 18603 tablet_bootstrap.cc:492] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:39.998054 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44df3acaa1a4f0d90f6553449d296a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.998780 12958 tablet_service.cc:1467] Processing CreateTablet for tablet cfbc588ca41e46b080296610f6739201 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.999066 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfbc588ca41e46b080296610f6739201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:39.999224 15331 tablet_service.cc:1467] Processing CreateTablet for tablet c44df3acaa1a4f0d90f6553449d296a8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:39.999431 18603 tablet_bootstrap.cc:654] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:39.999509 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44df3acaa1a4f0d90f6553449d296a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.000222 18331 tablet_bootstrap.cc:492] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:40.001190 18331 tablet_bootstrap.cc:654] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.001278 18605 tablet_bootstrap.cc:492] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:40.001968 12959 tablet_service.cc:1467] Processing CreateTablet for tablet c44df3acaa1a4f0d90f6553449d296a8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.002179 18605 tablet_bootstrap.cc:654] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.002205 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c44df3acaa1a4f0d90f6553449d296a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.002576 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 052117f3357f4577b80c0812b3ef2b05 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.002894 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052117f3357f4577b80c0812b3ef2b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.004144 18605 tablet_bootstrap.cc:492] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:40.004171 12567 tablet_service.cc:1467] Processing CreateTablet for tablet cfbc588ca41e46b080296610f6739201 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.004230 18605 ts_tablet_manager.cc:1367] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:40.004411 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfbc588ca41e46b080296610f6739201. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.004711 18605 raft_consensus.cc:348] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.004801 17717 raft_consensus.cc:1050] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c: attempting to promote NON_VOTER a6e94562bee149c2ace25672f0f9bf50 to VOTER
I20240501 14:09:40.004828 18605 raft_consensus.cc:374] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.004859 18605 raft_consensus.cc:729] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.004974 18605 consensus_queue.cc:260] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:39.995191 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 052117f3357f4577b80c0812b3ef2b05 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=bcdb87f99e6c476bb12dbdf5560084d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.005251 18605 ts_tablet_manager.cc:1398] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.005386 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 052117f3357f4577b80c0812b3ef2b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.005434 18605 tablet_bootstrap.cc:492] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:40.005362 17717 consensus_queue.cc:237] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:40.006512 18605 tablet_bootstrap.cc:654] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.007221 15353 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.007493 15222 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.007615 15422 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.009018 15422 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.009392 18213 tablet_bootstrap.cc:492] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:40.009907 12979 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.010462 18213 tablet_bootstrap.cc:654] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.010563 15425 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.012717 18605 tablet_bootstrap.cc:492] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:40.012809 18605 ts_tablet_manager.cc:1367] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:40.013123 18331 tablet_bootstrap.cc:492] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:40.013204 18331 ts_tablet_manager.cc:1367] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:40.013234 18213 tablet_bootstrap.cc:492] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:40.013305 18213 ts_tablet_manager.cc:1367] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:40.013267 18605 raft_consensus.cc:348] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.013366 18605 raft_consensus.cc:374] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.013396 18605 raft_consensus.cc:729] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.013509 18605 consensus_queue.cc:260] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.013689 18331 raft_consensus.cc:348] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.013697 18213 raft_consensus.cc:348] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.013783 18331 raft_consensus.cc:374] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.013790 18213 raft_consensus.cc:374] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.013797 18605 ts_tablet_manager.cc:1398] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.013819 18331 raft_consensus.cc:729] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.013820 18213 raft_consensus.cc:729] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.013933 18213 consensus_queue.cc:260] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.013940 18331 consensus_queue.cc:260] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.014194 18213 ts_tablet_manager.cc:1398] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:40.014219 18331 ts_tablet_manager.cc:1398] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.014389 18331 tablet_bootstrap.cc:492] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:40.016503 18213 tablet_bootstrap.cc:492] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:40.016654 18331 tablet_bootstrap.cc:654] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.017572 18213 tablet_bootstrap.cc:654] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.020331 18603 tablet_bootstrap.cc:492] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:40.020440 18603 ts_tablet_manager.cc:1367] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:40.020895 18603 raft_consensus.cc:348] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.020999 18603 raft_consensus.cc:374] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.021027 18603 raft_consensus.cc:729] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.021153 18603 consensus_queue.cc:260] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.021432 18603 ts_tablet_manager.cc:1398] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.021607 18603 tablet_bootstrap.cc:492] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:40.023329 18603 tablet_bootstrap.cc:654] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.025101 18603 tablet_bootstrap.cc:492] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:40.025178 18603 ts_tablet_manager.cc:1367] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:40.025560 18603 raft_consensus.cc:348] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.025652 18603 raft_consensus.cc:374] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.025678 18603 raft_consensus.cc:729] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.025797 18603 consensus_queue.cc:260] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.026048 18603 ts_tablet_manager.cc:1398] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:40.026440 14958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f"
dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51788
I20240501 14:09:40.026634 14958 raft_consensus.cc:595] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.027770 18331 tablet_bootstrap.cc:492] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:40.027865 18331 ts_tablet_manager.cc:1367] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:40.028323 18331 raft_consensus.cc:348] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.028434 18331 raft_consensus.cc:374] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.028491 18331 raft_consensus.cc:729] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.028609 18331 consensus_queue.cc:260] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.028913 18331 ts_tablet_manager.cc:1398] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:40.031265 18606 tablet_bootstrap.cc:492] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:40.032130 17415 consensus_queue.cc:1035] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.032761 18610 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.033098 18213 tablet_bootstrap.cc:492] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:40.033201 18213 ts_tablet_manager.cc:1367] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:40.033697 18213 raft_consensus.cc:348] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.033807 18213 raft_consensus.cc:374] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.033846 18213 raft_consensus.cc:729] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.033973 18213 consensus_queue.cc:260] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.034292 18213 ts_tablet_manager.cc:1398] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.034965 15222 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.035885 12979 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.035092 15353 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.039678 12443 catalog_manager.cc:5484] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.040547 18606 tablet_bootstrap.cc:654] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.043581 18606 tablet_bootstrap.cc:492] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:40.043707 18606 ts_tablet_manager.cc:1367] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:40.044193 18606 raft_consensus.cc:348] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.044330 18606 raft_consensus.cc:374] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.044389 18606 raft_consensus.cc:729] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.044533 18321 consensus_queue.cc:1035] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.044559 18606 consensus_queue.cc:260] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.044930 18606 ts_tablet_manager.cc:1398] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.047502 14958 raft_consensus.cc:1203] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:09:40.048709 15090 consensus_queue.cc:237] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:40.050832 12979 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.051914 17717 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.052233 15353 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.054087 17717 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.056114 17717 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.056677 15353 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.058487 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d521005dcbf74e4fb9a67fe9992ce540 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.059523 12443 catalog_manager.cc:5484] T d521005dcbf74e4fb9a67fe9992ce540 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.059729 18544 raft_consensus.cc:482] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.059832 18544 raft_consensus.cc:504] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.060251 18544 leader_election.cc:288] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:40.060883 12979 raft_consensus.cc:2932] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.061287 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbc588ca41e46b080296610f6739201" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:40.061393 12978 raft_consensus.cc:2454] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:40.061712 14893 leader_election.cc:302] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:40.062005 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbc588ca41e46b080296610f6739201" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:40.062142 12585 raft_consensus.cc:2454] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:40.063905 18544 raft_consensus.cc:2781] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.063987 18544 raft_consensus.cc:482] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.064030 18544 raft_consensus.cc:3037] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.065004 18544 raft_consensus.cc:504] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.065376 18544 leader_election.cc:288] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:40.065863 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet d521005dcbf74e4fb9a67fe9992ce540 with delete_type TABLET_DATA_TOMBSTONED (TS 66e2dc34330848c3b8c2dd15898d34c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:40.066185 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbc588ca41e46b080296610f6739201" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:40.066298 12585 raft_consensus.cc:3037] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.067255 12585 raft_consensus.cc:2454] T cfbc588ca41e46b080296610f6739201 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:40.067660 14893 leader_election.cc:302] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:40.067739 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:40.068097 18544 raft_consensus.cc:2781] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.068173 18544 raft_consensus.cc:686] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:40.068290 18544 consensus_queue.cc:237] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:40.069149 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbc588ca41e46b080296610f6739201" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:40.069281 12979 raft_consensus.cc:3037] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.072016 12440 catalog_manager.cc:5484] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.073318 12979 raft_consensus.cc:2454] T cfbc588ca41e46b080296610f6739201 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:40.073324 18579 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.073720 18579 tablet.cc:1620] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:40.074059 18580 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.074344 18580 tablet.cc:1620] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:40.076951 18541 raft_consensus.cc:482] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.077033 18541 raft_consensus.cc:504] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.077386 18541 leader_election.cc:288] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:40.077801 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052117f3357f4577b80c0812b3ef2b05" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:40.077917 12585 raft_consensus.cc:2454] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:40.077929 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052117f3357f4577b80c0812b3ef2b05" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:40.078058 15222 raft_consensus.cc:2454] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:40.078020 18578 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.078296 18578 tablet.cc:1620] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:40.078315 14891 leader_election.cc:302] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:40.078498 18541 raft_consensus.cc:2781] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.078548 18541 raft_consensus.cc:482] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.078605 18541 raft_consensus.cc:3037] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.079547 18541 raft_consensus.cc:504] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.079881 18541 leader_election.cc:288] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:40.080606 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052117f3357f4577b80c0812b3ef2b05" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:40.080722 15222 raft_consensus.cc:3037] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.081725 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "052117f3357f4577b80c0812b3ef2b05" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:40.081804 15222 raft_consensus.cc:2454] T 052117f3357f4577b80c0812b3ef2b05 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:40.081836 12585 raft_consensus.cc:3037] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.082621 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=9b7d4ae16a204ac5acb91c9dcbebc2a4] alter complete (version 2)
I20240501 14:09:40.082954 12585 raft_consensus.cc:2454] T 052117f3357f4577b80c0812b3ef2b05 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:40.083300 14891 leader_election.cc:302] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:40.083624 18541 raft_consensus.cc:2781] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.083716 18541 raft_consensus.cc:686] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:40.083845 18541 consensus_queue.cc:237] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.085867 12443 catalog_manager.cc:5484] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.088421 18637 tablet_replica.cc:330] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:40.088626 18637 raft_consensus.cc:2229] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.088841 18637 raft_consensus.cc:2258] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.090600 18637 ts_tablet_manager.cc:1875] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.093932 18637 ts_tablet_manager.cc:1888] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:40.094007 18637 log.cc:1196] T d521005dcbf74e4fb9a67fe9992ce540 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/d521005dcbf74e4fb9a67fe9992ce540
I20240501 14:09:40.096433 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet d521005dcbf74e4fb9a67fe9992ce540 (table rebalancer_extra_table_6 [id=f1a824378ee146df85971f1ebc4093ab]) successfully deleted
I20240501 14:09:40.096573 12848 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5957da39df64253a2b8d7f388308719"
dest_uuid: "4902d2609e6d44d089590a42e70356b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41436
I20240501 14:09:40.096714 12848 raft_consensus.cc:595] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.117516 12848 raft_consensus.cc:1203] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:40.118091 17415 raft_consensus.cc:979] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: : Instructing follower ab90e77f0027449bb3155980ca95593c to start an election
I20240501 14:09:40.118187 17415 raft_consensus.cc:1067] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Signalling peer ab90e77f0027449bb3155980ca95593c to start an election
I20240501 14:09:40.118961 15090 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a5957da39df64253a2b8d7f388308719"
dest_uuid: "ab90e77f0027449bb3155980ca95593c"
from {username='slave'} at 127.6.82.3:59095
I20240501 14:09:40.119136 15090 raft_consensus.cc:482] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.119192 15090 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.120429 15090 raft_consensus.cc:504] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.120872 15090 leader_election.cc:288] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 2 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:40.121205 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:40.121232 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:40.121225 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:40.121325 12848 raft_consensus.cc:3032] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.121344 12716 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.121344 12979 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.121377 12848 raft_consensus.cc:729] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:40.121526 12848 consensus_queue.cc:260] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.121740 12848 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.122315 12979 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 2.
I20240501 14:09:40.122315 12716 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 2.
I20240501 14:09:40.122570 12848 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 2.
I20240501 14:09:40.122645 15025 leader_election.cc:302] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:40.122826 15422 raft_consensus.cc:2781] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.122920 15422 raft_consensus.cc:686] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 2 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:40.123054 15422 consensus_queue.cc:237] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.124882 12441 catalog_manager.cc:5484] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 1 to 2, leader changed from 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 2 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.134405 18287 raft_consensus.cc:1050] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:40.134874 18287 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.136294 14958 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:40.136386 18238 raft_consensus.cc:1050] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:40.136463 12979 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:40.136463 15090 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:40.136785 18238 consensus_queue.cc:237] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.136898 18656 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:40.137084 17177 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:40.138295 14958 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.138404 12978 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.138485 12848 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.138684 15655 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.139042 15655 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.139256 18239 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.139744 17177 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:40.140862 17177 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.142369 18239 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.142998 15090 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.143599 12979 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.144347 12848 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.144694 12441 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 7 to 8, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.145449 14957 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.147287 12443 catalog_manager.cc:5484] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.148387 18279 raft_consensus.cc:1050] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:40.148759 18279 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:40.150264 12848 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:40.150671 17167 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.151141 12585 consensus_queue.cc:237] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.151911 15222 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:40.152412 16375 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.152513 14957 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:40.152858 15090 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:40.152978 12978 raft_consensus.cc:1261] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:40.153358 17272 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.153530 18238 consensus_queue.cc:1035] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.153599 12979 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.154381 16375 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.155432 18239 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.155802 14957 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.156962 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb336cffd30b447b880af97da6882d27 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.158098 14958 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.160650 12440 catalog_manager.cc:5484] T eb336cffd30b447b880af97da6882d27 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.161609 12978 raft_consensus.cc:2932] T eb336cffd30b447b880af97da6882d27 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.162119 16375 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.163053 15090 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.163868 15222 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.164044 12848 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.165329 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet eb336cffd30b447b880af97da6882d27 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:40.167198 18670 ts_tablet_manager.cc:902] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:40.167344 12439 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 5 to 6, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.168211 18670 tablet_copy_client.cc:308] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.7:34847
I20240501 14:09:40.168678 15242 tablet_copy_service.cc:140] P 66e2dc34330848c3b8c2dd15898d34c8: Received BeginTabletCopySession request for tablet e8226993467944dcb24d1efcda07538a from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:41919)
I20240501 14:09:40.168787 15242 tablet_copy_service.cc:161] P 66e2dc34330848c3b8c2dd15898d34c8: Beginning new tablet copy session on tablet e8226993467944dcb24d1efcda07538a from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:41919: session id = ab90e77f0027449bb3155980ca95593c-e8226993467944dcb24d1efcda07538a
I20240501 14:09:40.168995 18669 tablet_replica.cc:330] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:40.169163 18669 raft_consensus.cc:2229] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.169334 18669 raft_consensus.cc:2258] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.169688 15242 tablet_copy_source_session.cc:205] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:40.169929 18669 ts_tablet_manager.cc:1875] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.170657 18670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8226993467944dcb24d1efcda07538a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.172463 18669 ts_tablet_manager.cc:1888] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:40.172533 18669 log.cc:1196] T eb336cffd30b447b880af97da6882d27 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb336cffd30b447b880af97da6882d27
I20240501 14:09:40.173053 18670 tablet_copy_client.cc:783] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:40.173209 18670 tablet_copy_client.cc:647] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:40.173381 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet eb336cffd30b447b880af97da6882d27 (table rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]) successfully deleted
I20240501 14:09:40.173386 12978 consensus_queue.cc:237] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:40.174618 15222 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:40.174922 12848 raft_consensus.cc:1261] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:40.175001 18279 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.175146 18670 tablet_copy_client.cc:523] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:40.175302 18665 consensus_queue.cc:1035] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.176414 18670 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:40.176642 18279 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.177146 15222 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.177233 12848 raft_consensus.cc:2932] T 188ce319682147ddaad30aae7232a311 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.178922 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 188ce319682147ddaad30aae7232a311 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.179862 12439 catalog_manager.cc:5484] T 188ce319682147ddaad30aae7232a311 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 6 to 7, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.181385 18670 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:40.181639 18670 tablet_bootstrap.cc:492] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:40.181798 18670 ts_tablet_manager.cc:1367] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:09:40.182267 18670 raft_consensus.cc:348] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:40.182406 18670 raft_consensus.cc:729] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:40.182514 18670 consensus_queue.cc:260] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:40.182576 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet 188ce319682147ddaad30aae7232a311 with delete_type TABLET_DATA_TOMBSTONED (TS ab90e77f0027449bb3155980ca95593c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:40.182873 18670 ts_tablet_manager.cc:1398] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:40.183326 18676 tablet_replica.cc:330] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:40.183462 18676 raft_consensus.cc:2229] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.183658 18676 raft_consensus.cc:2258] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.183853 15242 tablet_copy_service.cc:342] P 66e2dc34330848c3b8c2dd15898d34c8: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-e8226993467944dcb24d1efcda07538a received from {username='slave'} at 127.6.82.6:41919
I20240501 14:09:40.183957 15242 tablet_copy_service.cc:433] P 66e2dc34330848c3b8c2dd15898d34c8: ending tablet copy session ab90e77f0027449bb3155980ca95593c-e8226993467944dcb24d1efcda07538a on tablet e8226993467944dcb24d1efcda07538a with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:40.184964 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:40.185065 18676 ts_tablet_manager.cc:1875] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.187343 18676 ts_tablet_manager.cc:1888] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:09:40.187484 18676 log.cc:1196] T 188ce319682147ddaad30aae7232a311 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/188ce319682147ddaad30aae7232a311
I20240501 14:09:40.188266 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 188ce319682147ddaad30aae7232a311 (table rebalancer_extra_table_4 [id=d55ee4f7c7fd4f978d63440f78e7a90d]) successfully deleted
I20240501 14:09:40.190536 18579 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.190876 18580 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.191278 18578 tablet.cc:1722] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.195935 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=9b7d4ae16a204ac5acb91c9dcbebc2a4] alter complete (version 3)
I20240501 14:09:40.201457 17925 raft_consensus.cc:1050] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8: attempting to promote NON_VOTER a6e94562bee149c2ace25672f0f9bf50 to VOTER
I20240501 14:09:40.201821 17925 consensus_queue.cc:237] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:40.203071 12978 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.203094 14957 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.203512 16475 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.203667 18520 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.203779 15090 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.204134 18520 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.204453 18656 raft_consensus.cc:1050] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:40.204845 18656 consensus_queue.cc:237] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.206094 12849 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.206380 15090 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.206715 18287 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.206887 18286 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.207291 12979 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.207746 18287 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.209640 18287 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:40.210198 16475 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.210238 12848 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:40.212810 14957 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.213534 12439 catalog_manager.cc:5484] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.214672 12443 catalog_manager.cc:5484] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:40.215349 12978 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.217357 12980 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:40.219137 15353 consensus_queue.cc:237] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:40.220395 12980 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.221005 14957 raft_consensus.cc:1261] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.221408 18059 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.221716 18059 consensus_queue.cc:1035] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.224423 18059 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.225054 12978 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.225625 15089 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:40.226322 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 565e6fb440dc4b00b3e2d0e1936889c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.228675 12439 catalog_manager.cc:5484] T 565e6fb440dc4b00b3e2d0e1936889c8 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.231621 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet 565e6fb440dc4b00b3e2d0e1936889c8 with delete_type TABLET_DATA_TOMBSTONED (TS ab90e77f0027449bb3155980ca95593c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:40.231892 18676 tablet_replica.cc:330] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:40.232036 18676 raft_consensus.cc:2229] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.232110 18286 raft_consensus.cc:1050] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER a6e94562bee149c2ace25672f0f9bf50 to VOTER
I20240501 14:09:40.232156 18676 pending_rounds.cc:62] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Trying to abort 1 pending ops.
I20240501 14:09:40.232223 18676 pending_rounds.cc:68] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022889288505761792 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "565e6fb440dc4b00b3e2d0e1936889c8" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } } }
I20240501 14:09:40.232365 18676 raft_consensus.cc:2894] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:09:40.232429 18676 raft_consensus.cc:2866] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:40.232484 18676 raft_consensus.cc:2258] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.232471 18286 consensus_queue.cc:237] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:40.233219 18676 ts_tablet_manager.cc:1875] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.233243 14957 raft_consensus.cc:2932] T 565e6fb440dc4b00b3e2d0e1936889c8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.233937 14958 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.233933 15222 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.234323 17177 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.234793 17177 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.235891 18676 ts_tablet_manager.cc:1888] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:40.236016 18676 log.cc:1196] T 565e6fb440dc4b00b3e2d0e1936889c8 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/565e6fb440dc4b00b3e2d0e1936889c8
I20240501 14:09:40.236595 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 565e6fb440dc4b00b3e2d0e1936889c8 (table rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]) successfully deleted
I20240501 14:09:40.238279 12978 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.238705 17177 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.240622 18656 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.241138 12978 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.242678 15222 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.244971 14957 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.245005 12438 catalog_manager.cc:5484] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 2 to 3, a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.250659 12716 consensus_queue.cc:237] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:40.251907 12978 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.252252 14957 raft_consensus.cc:1261] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.252251 18656 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.252640 18656 consensus_queue.cc:1035] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.254225 18656 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.255748 14957 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.256109 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee9a1e9b13f546cc81aae6d729c8974f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.256681 12978 raft_consensus.cc:2932] T ee9a1e9b13f546cc81aae6d729c8974f P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } } }
I20240501 14:09:40.257417 12441 catalog_manager.cc:5484] T ee9a1e9b13f546cc81aae6d729c8974f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.261497 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet ee9a1e9b13f546cc81aae6d729c8974f with delete_type TABLET_DATA_TOMBSTONED (TS 66e2dc34330848c3b8c2dd15898d34c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:40.261723 18637 tablet_replica.cc:330] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:40.261850 18637 raft_consensus.cc:2229] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.262001 18637 raft_consensus.cc:2258] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.262580 18637 ts_tablet_manager.cc:1875] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.264503 18637 ts_tablet_manager.cc:1888] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:40.264571 18637 log.cc:1196] T ee9a1e9b13f546cc81aae6d729c8974f P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/ee9a1e9b13f546cc81aae6d729c8974f
I20240501 14:09:40.265304 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet ee9a1e9b13f546cc81aae6d729c8974f (table rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]) successfully deleted
I20240501 14:09:40.294544 18544 raft_consensus.cc:979] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: : Instructing follower e6ab42bc38694f7f9fc8bfcc0a1483b7 to start an election
I20240501 14:09:40.294677 18544 raft_consensus.cc:1067] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Signalling peer e6ab42bc38694f7f9fc8bfcc0a1483b7 to start an election
I20240501 14:09:40.295358 12716 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
from {username='slave'} at 127.6.82.5:37435
I20240501 14:09:40.295506 12716 raft_consensus.cc:482] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.295571 12716 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.296550 12716 raft_consensus.cc:504] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.297048 12716 leader_election.cc:288] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:40.297905 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:40.298043 12848 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.298085 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:40.298209 15089 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.298624 18286 raft_consensus.cc:1050] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:40.299057 18286 consensus_queue.cc:237] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.299207 12848 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:40.299207 15089 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:40.299594 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:40.297683 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df57dee35f924801a23bd9b5c547a55f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:40.299736 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:40.299760 14957 raft_consensus.cc:3032] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.299809 14957 raft_consensus.cc:729] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:40.300232 18396 raft_consensus.cc:482] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.300338 18396 raft_consensus.cc:504] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.300036 14957 consensus_queue.cc:260] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.300781 18396 leader_election.cc:288] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:40.300951 12848 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.300995 15353 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.301384 18287 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.301616 17177 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.301770 12980 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.302109 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44df3acaa1a4f0d90f6553449d296a8" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:40.302120 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44df3acaa1a4f0d90f6553449d296a8" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:40.302227 15353 raft_consensus.cc:2454] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:40.302390 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet cbf3fa8e0dec4ceb9057f70480aa5318
I20240501 14:09:40.301401 12653 leader_election.cc:302] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:40.302701 18287 raft_consensus.cc:2781] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.302842 18287 raft_consensus.cc:686] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:40.302973 18287 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.303112 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf3fa8e0dec4ceb9057f70480aa5318 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:40.303386 15157 leader_election.cc:302] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:40.303606 18396 raft_consensus.cc:2781] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.303591 18287 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.303674 18396 raft_consensus.cc:482] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.303726 18396 raft_consensus.cc:3037] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.303700 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf3fa8e0dec4ceb9057f70480aa5318 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:40.304302 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf3fa8e0dec4ceb9057f70480aa5318 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:40.302618 14957 raft_consensus.cc:3037] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.304869 18396 raft_consensus.cc:504] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.305222 18396 leader_election.cc:288] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:40.305392 14957 raft_consensus.cc:2454] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:40.305598 18669 tablet_replica.cc:330] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:40.305739 18669 raft_consensus.cc:2229] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.305862 18669 raft_consensus.cc:2258] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.306488 18669 ts_tablet_manager.cc:1875] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.307938 18520 raft_consensus.cc:1050] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: attempting to promote NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 to VOTER
I20240501 14:09:40.307960 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44df3acaa1a4f0d90f6553449d296a8" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:40.308288 15351 raft_consensus.cc:3037] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.308424 18520 consensus_queue.cc:237] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.308521 18692 tablet_replica.cc:330] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:40.308671 18669 ts_tablet_manager.cc:1888] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.308673 18692 raft_consensus.cc:2229] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:40.308734 18669 log.cc:1196] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/cbf3fa8e0dec4ceb9057f70480aa5318
I20240501 14:09:40.309001 18669 ts_tablet_manager.cc:1909] T cbf3fa8e0dec4ceb9057f70480aa5318 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:40.308952 18286 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:40.309242 15351 raft_consensus.cc:2454] T c44df3acaa1a4f0d90f6553449d296a8 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:40.309887 12978 raft_consensus.cc:2454] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:40.309906 12980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44df3acaa1a4f0d90f6553449d296a8" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:40.310474 12980 raft_consensus.cc:3037] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.311045 18692 raft_consensus.cc:2258] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.311061 12979 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:40.311291 18693 tablet_replica.cc:330] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:40.311456 18693 raft_consensus.cc:2229] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.311478 12980 raft_consensus.cc:2454] T c44df3acaa1a4f0d90f6553449d296a8 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:40.311429 12439 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 1 to 2, leader changed from 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.311580 18693 raft_consensus.cc:2258] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.311961 15156 leader_election.cc:302] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:40.312273 18693 ts_tablet_manager.cc:1875] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.312263 12848 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:40.312577 18396 raft_consensus.cc:2781] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.312739 12585 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.312857 18692 ts_tablet_manager.cc:1875] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.313259 18396 raft_consensus.cc:686] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:40.313412 18396 consensus_queue.cc:237] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:40.313838 18520 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.315176 18692 ts_tablet_manager.cc:1888] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.315251 18692 log.cc:1196] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/cbf3fa8e0dec4ceb9057f70480aa5318
I20240501 14:09:40.315603 18692 ts_tablet_manager.cc:1909] T cbf3fa8e0dec4ceb9057f70480aa5318 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:40.315517 12438 catalog_manager.cc:5484] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:40.316198 15089 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.316558 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table_new_name [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]) successfully deleted
I20240501 14:09:40.316911 18520 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.317044 12716 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEARNER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.317051 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table_new_name [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]) successfully deleted
I20240501 14:09:40.318084 12439 catalog_manager.cc:5484] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.318639 18520 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.321177 16475 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:40.321486 15089 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:40.321910 18693 ts_tablet_manager.cc:1888] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.322001 18693 log.cc:1196] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/cbf3fa8e0dec4ceb9057f70480aa5318
I20240501 14:09:40.322314 18693 ts_tablet_manager.cc:1909] T cbf3fa8e0dec4ceb9057f70480aa5318 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:40.324985 12441 catalog_manager.cc:5484] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.325002 12585 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:40.325531 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet cbf3fa8e0dec4ceb9057f70480aa5318 (table rebalancer_dynamic_table_new_name [id=9b7d4ae16a204ac5acb91c9dcbebc2a4]) successfully deleted
I20240501 14:09:40.326341 12716 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:40.328668 15353 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:40.332679 18706 ts_tablet_manager.cc:902] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:40.333148 18706 tablet_copy_client.cc:308] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:40.333595 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet 63a76e89d58d48ba9d1b3b98ef6c172d from peer f23f426391724d6688aafffe7ee017d8 ({username='slave'} at 127.6.82.8:45593)
I20240501 14:09:40.333710 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet 63a76e89d58d48ba9d1b3b98ef6c172d from peer f23f426391724d6688aafffe7ee017d8 at {username='slave'} at 127.6.82.8:45593: session id = f23f426391724d6688aafffe7ee017d8-63a76e89d58d48ba9d1b3b98ef6c172d
I20240501 14:09:40.334635 12999 tablet_copy_source_session.cc:205] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:40.335709 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a76e89d58d48ba9d1b3b98ef6c172d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.337710 18706 tablet_copy_client.cc:783] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:40.337864 18706 tablet_copy_client.cc:647] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:40.339745 18706 tablet_copy_client.cc:523] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:40.341203 18706 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:40.359478 12716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40194
I20240501 14:09:40.359627 12716 raft_consensus.cc:595] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.367132 18706 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:40.367484 18706 tablet_bootstrap.cc:492] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: Bootstrap complete.
I20240501 14:09:40.367789 18706 ts_tablet_manager.cc:1367] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240501 14:09:40.368652 18706 raft_consensus.cc:348] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:40.368928 18706 raft_consensus.cc:729] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: LEARNER
I20240501 14:09:40.369109 18706 consensus_queue.cc:260] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:09:40.373180 18706 ts_tablet_manager.cc:1398] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:40.374572 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session f23f426391724d6688aafffe7ee017d8-63a76e89d58d48ba9d1b3b98ef6c172d received from {username='slave'} at 127.6.82.8:45593
I20240501 14:09:40.374667 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session f23f426391724d6688aafffe7ee017d8-63a76e89d58d48ba9d1b3b98ef6c172d on tablet 63a76e89d58d48ba9d1b3b98ef6c172d with peer f23f426391724d6688aafffe7ee017d8
I20240501 14:09:40.375156 15353 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29"
dest_uuid: "f23f426391724d6688aafffe7ee017d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37834
I20240501 14:09:40.375286 15353 raft_consensus.cc:595] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.394860 12716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40216
I20240501 14:09:40.395010 12716 raft_consensus.cc:595] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.396539 12585 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:40.397398 18321 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.404964 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:40.411866 15068 tablet_service.cc:1467] Processing CreateTablet for tablet e5d4b8858d2a43ac836505cae472f003 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]), partition=
I20240501 14:09:40.412132 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d4b8858d2a43ac836505cae472f003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.412129 12698 tablet_service.cc:1467] Processing CreateTablet for tablet e5d4b8858d2a43ac836505cae472f003 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]), partition=
I20240501 14:09:40.412129 15202 tablet_service.cc:1467] Processing CreateTablet for tablet e5d4b8858d2a43ac836505cae472f003 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]), partition=
I20240501 14:09:40.412390 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d4b8858d2a43ac836505cae472f003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.412390 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d4b8858d2a43ac836505cae472f003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.414502 18603 tablet_bootstrap.cc:492] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:40.415187 18767 tablet_bootstrap.cc:492] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:40.415550 18603 tablet_bootstrap.cc:654] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.415714 18766 tablet_bootstrap.cc:492] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:40.416743 18766 tablet_bootstrap.cc:654] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.418356 18767 tablet_bootstrap.cc:654] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.419611 18766 tablet_bootstrap.cc:492] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:40.419701 18766 ts_tablet_manager.cc:1367] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:40.420152 18766 raft_consensus.cc:348] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.420267 18766 raft_consensus.cc:374] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.420305 18766 raft_consensus.cc:729] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.420889 18767 tablet_bootstrap.cc:492] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:40.420976 18767 ts_tablet_manager.cc:1367] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:40.421502 18767 raft_consensus.cc:348] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.421617 18767 raft_consensus.cc:374] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.421651 18767 raft_consensus.cc:729] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.421767 18767 consensus_queue.cc:260] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.421994 18766 consensus_queue.cc:260] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.422395 18603 tablet_bootstrap.cc:492] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:40.422499 18603 ts_tablet_manager.cc:1367] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:40.423130 18603 raft_consensus.cc:348] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.423251 18603 raft_consensus.cc:374] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.423288 18603 raft_consensus.cc:729] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.423429 18603 consensus_queue.cc:260] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.423678 18603 ts_tablet_manager.cc:1398] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.424520 18766 ts_tablet_manager.cc:1398] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:40.424834 18767 ts_tablet_manager.cc:1398] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:40.426596 12716 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:40.427026 18321 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.429680 15222 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:40.430543 15878 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.435410 12585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce"
dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34734
I20240501 14:09:40.435541 12585 raft_consensus.cc:595] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.447660 12716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40226
I20240501 14:09:40.447811 12716 raft_consensus.cc:595] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.466902 15422 raft_consensus.cc:482] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.467059 15422 raft_consensus.cc:504] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.467603 15422 leader_election.cc:288] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:40.468063 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d4b8858d2a43ac836505cae472f003" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:40.468108 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d4b8858d2a43ac836505cae472f003" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:40.468216 12716 raft_consensus.cc:2454] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:40.468256 15222 raft_consensus.cc:2454] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 0.
I20240501 14:09:40.468578 15023 leader_election.cc:302] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:40.468860 15422 raft_consensus.cc:2781] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.468899 15422 raft_consensus.cc:482] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.468924 15422 raft_consensus.cc:3037] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.469910 15422 raft_consensus.cc:504] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.470252 15422 leader_election.cc:288] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:40.470623 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d4b8858d2a43ac836505cae472f003" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:40.470786 15222 raft_consensus.cc:3037] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.471107 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d4b8858d2a43ac836505cae472f003" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:40.471256 12716 raft_consensus.cc:3037] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.471889 15222 raft_consensus.cc:2454] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:40.472388 12716 raft_consensus.cc:2454] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 1.
I20240501 14:09:40.473685 15023 leader_election.cc:302] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:40.473874 15422 raft_consensus.cc:2781] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.473956 15422 raft_consensus.cc:686] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:40.474095 15422 consensus_queue.cc:237] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.475981 12439 catalog_manager.cc:5484] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 0 to 1, leader changed from <none> to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.492592 18544 consensus_queue.cc:1035] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.495704 18544 consensus_queue.cc:1035] T cfbc588ca41e46b080296610f6739201 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.507632 12716 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:40.509146 12978 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:40.509152 14957 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:40.509387 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abac52c109174021ac11360b0d22ccfa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:40.509618 18656 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.509814 18287 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.511687 15422 raft_consensus.cc:1050] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c: attempting to promote NON_VOTER 4902d2609e6d44d089590a42e70356b2 to VOTER
I20240501 14:09:40.511787 18287 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:40.512145 15422 consensus_queue.cc:237] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:40.512117 12978 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:40.513700 15222 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.513692 15353 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.514129 15422 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.514580 12848 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:40.515281 14957 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:40.516255 12443 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:40.518075 15425 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.519872 15425 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:40.521310 15425 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.521672 15222 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.524034 15353 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.524308 12443 catalog_manager.cc:5484] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.525919 15655 raft_consensus.cc:979] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: : Instructing follower f23f426391724d6688aafffe7ee017d8 to start an election
I20240501 14:09:40.526019 15655 raft_consensus.cc:1067] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Signalling peer f23f426391724d6688aafffe7ee017d8 to start an election
I20240501 14:09:40.527288 15353 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce"
dest_uuid: "f23f426391724d6688aafffe7ee017d8"
from {username='slave'} at 127.6.82.1:52105
I20240501 14:09:40.527431 15353 raft_consensus.cc:482] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.527504 15353 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.528412 15353 raft_consensus.cc:504] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.528890 15353 leader_election.cc:288] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:40.529558 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:40.529691 12585 raft_consensus.cc:3032] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.529726 12585 raft_consensus.cc:729] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:40.529862 12585 consensus_queue.cc:260] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.529800 12849 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.530056 12585 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.530974 12585 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:40.531337 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:40.531463 12848 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.531663 15089 consensus_queue.cc:237] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:40.532411 12848 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:40.532909 12848 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.533016 15353 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:40.533023 15288 leader_election.cc:302] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:40.533277 18059 raft_consensus.cc:2781] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.533365 18059 raft_consensus.cc:686] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 2 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:40.533358 17717 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.533506 18059 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.533767 17717 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:40.535326 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7693e3e204634676a758726bb6d3d2ce" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:40.535467 15222 raft_consensus.cc:3037] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.535710 15422 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.536477 15222 raft_consensus.cc:2454] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:40.537114 12443 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 1 to 2, leader changed from 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 2 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.537503 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d6a7ed55367493e9edd0461d6da6313 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.538898 15353 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.539036 12441 catalog_manager.cc:5484] T 3d6a7ed55367493e9edd0461d6da6313 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New cstate: current_term: 1 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.540966 12978 consensus_queue.cc:237] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:40.540971 12848 raft_consensus.cc:2932] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:40.542681 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d6a7ed55367493e9edd0461d6da6313 with delete_type TABLET_DATA_TOMBSTONED (TS 66e2dc34330848c3b8c2dd15898d34c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:40.542969 18637 tablet_replica.cc:330] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:40.542695 15222 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.543123 18637 raft_consensus.cc:2229] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.543263 18637 raft_consensus.cc:2258] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.544718 14957 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.544760 17167 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:40.545362 18665 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:09:40.545778 12916 consensus_peers.cc:485] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2639132e2d04ce480edc800214f25be. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:40.546351 18637 ts_tablet_manager.cc:1875] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.546823 16375 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.547907 12848 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:40.548334 18610 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.548622 18637 ts_tablet_manager.cc:1888] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:40.548678 18637 log.cc:1196] T 3d6a7ed55367493e9edd0461d6da6313 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/3d6a7ed55367493e9edd0461d6da6313
I20240501 14:09:40.549539 15222 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.549891 12439 catalog_manager.cc:5484] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.552657 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 3d6a7ed55367493e9edd0461d6da6313 (table rebalancer_extra_table_8 [id=7e020573285f4f9987d35c745f78316d]) successfully deleted
I20240501 14:09:40.555922 14957 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.581486 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:40.581859 17177 raft_consensus.cc:979] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: : Instructing follower ab90e77f0027449bb3155980ca95593c to start an election
I20240501 14:09:40.581954 17177 raft_consensus.cc:1067] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Signalling peer ab90e77f0027449bb3155980ca95593c to start an election
I20240501 14:09:40.583734 15089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d"
dest_uuid: "ab90e77f0027449bb3155980ca95593c"
from {username='slave'} at 127.6.82.2:41317
I20240501 14:09:40.583859 15089 raft_consensus.cc:482] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.583925 15089 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.584926 15089 raft_consensus.cc:504] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.585362 15089 leader_election.cc:288] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 2 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:40.585675 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:40.585795 12978 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.586928 12978 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 2.
I20240501 14:09:40.587164 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:40.587189 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "ab90e77f0027449bb3155980ca95593c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:40.587289 12716 raft_consensus.cc:3032] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.587302 14957 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.587337 12716 raft_consensus.cc:729] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:40.587476 12716 consensus_queue.cc:260] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.587659 12716 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.588168 14957 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 2.
I20240501 14:09:40.588558 12716 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab90e77f0027449bb3155980ca95593c in term 2.
I20240501 14:09:40.588903 15023 leader_election.cc:302] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:40.589053 18610 raft_consensus.cc:2781] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.589124 18610 raft_consensus.cc:686] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 2 LEADER]: Becoming Leader. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Running, Role: LEADER
I20240501 14:09:40.589242 18610 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.590276 12978 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:40.590651 18610 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.592162 12441 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c reported cstate change: term changed from 1 to 2, leader changed from e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) to ab90e77f0027449bb3155980ca95593c (127.6.82.6). New cstate: current_term: 2 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.593520 12716 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:40.594060 15425 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:40.597043 15222 raft_consensus.cc:1261] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.597581 12717 raft_consensus.cc:1261] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.597944 15425 consensus_queue.cc:1035] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.598306 15425 consensus_queue.cc:1035] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.598415 18826 consensus_queue.cc:1035] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.605799 12439 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]). Expected version 1 got 0
I20240501 14:09:40.606518 18826 consensus_queue.cc:1035] T 052117f3357f4577b80c0812b3ef2b05 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.608522 12439 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]). Expected version 1 got 0
I20240501 14:09:40.610379 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]). Expected version 1 got 0
I20240501 14:09:40.611621 18874 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.611707 12438 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047]). Expected version 1 got 0
I20240501 14:09:40.611974 18874 tablet.cc:1620] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:40.612665 18875 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.613054 18875 tablet.cc:1620] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:40.613910 18877 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:40.614230 18877 tablet.cc:1620] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:40.619693 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047] alter complete (version 1)
I20240501 14:09:40.628890 17415 raft_consensus.cc:1050] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER 8a13c028ba8a4a659b74af6c265df95e to VOTER
I20240501 14:09:40.629314 17415 consensus_queue.cc:237] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:40.630637 14957 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.631026 12585 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.631108 12716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40260
I20240501 14:09:40.631254 12716 raft_consensus.cc:595] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.631345 15878 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.631537 16502 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.631824 12716 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:40.632264 17415 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:40.635929 16502 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.636317 12716 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.636735 12585 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.638242 14957 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.639374 12443 catalog_manager.cc:5484] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 4 to 5, 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:40.651392 15353 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29"
dest_uuid: "f23f426391724d6688aafffe7ee017d8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37858
I20240501 14:09:40.651533 15353 raft_consensus.cc:595] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.662648 12717 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40288
I20240501 14:09:40.662838 12717 raft_consensus.cc:595] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.667894 18286 raft_consensus.cc:979] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: : Instructing follower 4902d2609e6d44d089590a42e70356b2 to start an election
I20240501 14:09:40.667985 18286 raft_consensus.cc:1067] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Signalling peer 4902d2609e6d44d089590a42e70356b2 to start an election
I20240501 14:09:40.668396 12848 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd"
dest_uuid: "4902d2609e6d44d089590a42e70356b2"
from {username='slave'} at 127.6.82.2:50703
I20240501 14:09:40.668545 12848 raft_consensus.cc:482] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.668615 12848 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.669757 12848 raft_consensus.cc:504] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.670202 12848 leader_election.cc:288] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:40.670779 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:40.670778 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:40.670853 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e476bc46420d992a43bf3b4c73fd" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:40.670915 15089 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.670928 12978 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.670967 12717 raft_consensus.cc:3032] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.671007 12717 raft_consensus.cc:729] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:40.671164 12717 consensus_queue.cc:260] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.671376 12717 raft_consensus.cc:3037] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.671789 12978 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 2.
I20240501 14:09:40.671979 15089 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 2.
I20240501 14:09:40.672257 12784 leader_election.cc:302] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:40.672395 12717 raft_consensus.cc:2454] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 2.
I20240501 14:09:40.672466 18321 raft_consensus.cc:2781] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.672565 18321 raft_consensus.cc:686] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 2 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:40.672701 18321 consensus_queue.cc:237] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:40.674870 12441 catalog_manager.cc:5484] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 1 to 2, leader changed from e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 2 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:40.699738 14957 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:40.700245 18286 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.703704 15089 raft_consensus.cc:1203] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:40.713706 18656 raft_consensus.cc:979] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: : Instructing follower f23f426391724d6688aafffe7ee017d8 to start an election
I20240501 14:09:40.713809 18656 raft_consensus.cc:1067] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Signalling peer f23f426391724d6688aafffe7ee017d8 to start an election
I20240501 14:09:40.714452 15351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd"
dest_uuid: "f23f426391724d6688aafffe7ee017d8"
from {username='slave'} at 127.6.82.2:36499
I20240501 14:09:40.714581 15351 raft_consensus.cc:482] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.714646 15351 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.715461 12848 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:40.715595 15351 raft_consensus.cc:504] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.715801 15353 consensus_queue.cc:237] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:40.715883 18656 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.716034 15351 leader_election.cc:288] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:40.716384 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:40.716498 12717 raft_consensus.cc:3032] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.716540 12717 raft_consensus.cc:729] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:40.716516 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:40.716634 12848 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.716682 12717 consensus_queue.cc:260] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.716835 12717 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.717607 12848 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:40.717738 12717 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:40.718396 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6628290de33e4aba8ccd1651d1d054dd" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:40.718520 12978 raft_consensus.cc:3037] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.718614 15221 raft_consensus.cc:1261] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:09:40.718636 15289 consensus_peers.cc:485] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bd8d72845814f17961145fd8d4d0bbd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:40.719547 12978 raft_consensus.cc:2454] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:40.719691 15089 raft_consensus.cc:1261] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:40.719849 15288 leader_election.cc:302] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:40.721108 15089 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:40.721508 18287 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.722184 18520 raft_consensus.cc:2781] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.722285 18520 raft_consensus.cc:686] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 2 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:40.722404 18520 consensus_queue.cc:237] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:40.723095 18520 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:40.725054 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:40.725345 17925 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:40.725723 16475 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.725692 15221 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.727538 15089 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.728603 12439 catalog_manager.cc:5484] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:09:40.732354 18874 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.732698 18874 tablet.cc:1620] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:40.734153 12440 catalog_manager.cc:5484] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 1 to 2, leader changed from e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 2 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.734850 18875 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.735190 18875 tablet.cc:1620] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:40.740207 18877 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:40.740527 18877 tablet.cc:1620] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:40.743254 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=e1c76b8df8c64dbca3bd54f2b10f8047] alter complete (version 2)
I20240501 14:09:40.748378 17925 raft_consensus.cc:979] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: : Instructing follower e6ab42bc38694f7f9fc8bfcc0a1483b7 to start an election
I20240501 14:09:40.748507 17925 raft_consensus.cc:1067] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Signalling peer e6ab42bc38694f7f9fc8bfcc0a1483b7 to start an election
I20240501 14:09:40.749042 12717 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
from {username='slave'} at 127.6.82.8:35333
I20240501 14:09:40.749183 12717 raft_consensus.cc:482] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:40.749253 12717 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.750237 12717 raft_consensus.cc:504] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.750979 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:40.751121 12585 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.751132 12717 leader_election.cc:288] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:40.751515 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:40.751534 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7130a0cb5bca433d8eb682e05b7a8f29" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:40.751621 15353 raft_consensus.cc:3032] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:40.751636 15089 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.751657 15353 raft_consensus.cc:729] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:40.751819 15353 consensus_queue.cc:260] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.752025 15353 raft_consensus.cc:3037] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:40.752281 12585 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:40.752377 15089 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:40.752955 15353 raft_consensus.cc:2454] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:40.753222 12654 leader_election.cc:302] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:40.753515 18287 raft_consensus.cc:2781] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:40.753610 18287 raft_consensus.cc:686] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:40.753751 18287 consensus_queue.cc:237] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:40.756037 12440 catalog_manager.cc:5484] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 1 to 2, leader changed from f23f426391724d6688aafffe7ee017d8 (127.6.82.8) to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.771103 12849 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc"
dest_uuid: "4902d2609e6d44d089590a42e70356b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41486
I20240501 14:09:40.771238 12849 raft_consensus.cc:595] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:09:40.828022 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:40.835036 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 28de192a12d14aa1a287b1473c4ea02d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.835379 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de192a12d14aa1a287b1473c4ea02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.835976 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 5f213a1449f940b7884e357f84448e27 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.836256 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f213a1449f940b7884e357f84448e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.836735 15201 tablet_service.cc:1467] Processing CreateTablet for tablet cee040fd7ece4b8d8af328f9205dae20 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.836977 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee040fd7ece4b8d8af328f9205dae20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.837028 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 28de192a12d14aa1a287b1473c4ea02d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.837311 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de192a12d14aa1a287b1473c4ea02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.837968 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 5f213a1449f940b7884e357f84448e27 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.838186 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f213a1449f940b7884e357f84448e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.838701 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 5f213a1449f940b7884e357f84448e27 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.838992 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f213a1449f940b7884e357f84448e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.839120 14938 tablet_service.cc:1467] Processing CreateTablet for tablet cee040fd7ece4b8d8af328f9205dae20 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.839354 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee040fd7ece4b8d8af328f9205dae20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.839479 15068 tablet_service.cc:1467] Processing CreateTablet for tablet cee040fd7ece4b8d8af328f9205dae20 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.839730 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee040fd7ece4b8d8af328f9205dae20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.841228 18603 tablet_bootstrap.cc:492] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:40.841447 19025 tablet_bootstrap.cc:492] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:40.841928 18766 tablet_bootstrap.cc:492] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:40.836203 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 28de192a12d14aa1a287b1473c4ea02d (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=eeca6b5e52b9455e9c61f09aa9868a54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:40.842198 18603 tablet_bootstrap.cc:654] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.842279 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de192a12d14aa1a287b1473c4ea02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.842826 18766 tablet_bootstrap.cc:654] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.844026 19025 tablet_bootstrap.cc:654] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.844106 19027 tablet_bootstrap.cc:492] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:40.845223 19027 tablet_bootstrap.cc:654] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.846119 18603 tablet_bootstrap.cc:492] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:40.846218 18603 ts_tablet_manager.cc:1367] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:09:40.846323 18766 tablet_bootstrap.cc:492] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:40.846397 18766 ts_tablet_manager.cc:1367] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:40.846421 19025 tablet_bootstrap.cc:492] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:40.846498 19025 ts_tablet_manager.cc:1367] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:40.846702 18603 raft_consensus.cc:348] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.846863 18603 raft_consensus.cc:374] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.846896 18603 raft_consensus.cc:729] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.846843 18766 raft_consensus.cc:348] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.846952 18766 raft_consensus.cc:374] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.846987 18766 raft_consensus.cc:729] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.846951 19025 raft_consensus.cc:348] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.847020 18603 consensus_queue.cc:260] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.847064 19025 raft_consensus.cc:374] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.847004 12849 consensus_queue.cc:237] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:40.847096 19025 raft_consensus.cc:729] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.847095 18766 consensus_queue.cc:260] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.847203 19025 consensus_queue.cc:260] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.847396 18766 ts_tablet_manager.cc:1398] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:40.847862 18603 ts_tablet_manager.cc:1398] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:40.848019 18603 tablet_bootstrap.cc:492] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:40.848268 19025 ts_tablet_manager.cc:1398] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:40.849052 15221 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:40.849179 18603 tablet_bootstrap.cc:654] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.849431 12717 raft_consensus.cc:1261] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:40.850528 15878 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.850956 18603 tablet_bootstrap.cc:492] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:40.851044 18603 ts_tablet_manager.cc:1367] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:09:40.851657 18603 raft_consensus.cc:348] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.851763 18603 raft_consensus.cc:374] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.851799 18603 raft_consensus.cc:729] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.851806 17415 consensus_queue.cc:1035] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:40.851944 18603 consensus_queue.cc:260] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.852321 18603 ts_tablet_manager.cc:1398] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.852483 18603 tablet_bootstrap.cc:492] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:40.853332 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:40.853863 18603 tablet_bootstrap.cc:654] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.854823 15878 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.859228 15221 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.859359 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 444c1a9d0e6943dc880161ae320673b6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:40.860085 12440 catalog_manager.cc:5484] T 444c1a9d0e6943dc880161ae320673b6 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 5 to 7, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.860644 19026 tablet_bootstrap.cc:492] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:40.861609 19026 tablet_bootstrap.cc:654] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.862915 19027 tablet_bootstrap.cc:492] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:40.863018 19027 ts_tablet_manager.cc:1367] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240501 14:09:40.863541 19027 raft_consensus.cc:348] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.863646 19027 raft_consensus.cc:374] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.863678 19027 raft_consensus.cc:729] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.863796 19027 consensus_queue.cc:260] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.864074 19027 ts_tablet_manager.cc:1398] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:40.864256 19027 tablet_bootstrap.cc:492] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:40.865118 19027 tablet_bootstrap.cc:654] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.865284 12717 raft_consensus.cc:2932] T 444c1a9d0e6943dc880161ae320673b6 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:40.868333 18603 tablet_bootstrap.cc:492] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:40.868490 18603 ts_tablet_manager.cc:1367] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:40.868767 18874 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.869104 18603 raft_consensus.cc:348] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.869288 18603 raft_consensus.cc:374] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.869374 18603 raft_consensus.cc:729] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.869565 18603 consensus_queue.cc:260] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.869908 18603 ts_tablet_manager.cc:1398] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.871235 18877 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.872958 19027 tablet_bootstrap.cc:492] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:40.873070 19027 ts_tablet_manager.cc:1367] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:40.873533 19027 raft_consensus.cc:348] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.873664 19027 raft_consensus.cc:374] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.873704 19027 raft_consensus.cc:729] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.873705 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 444c1a9d0e6943dc880161ae320673b6 with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:40.873907 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=e1c76b8df8c64dbca3bd54f2b10f8047] alter complete (version 3)
I20240501 14:09:40.874094 19036 tablet_replica.cc:330] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:40.874158 19027 consensus_queue.cc:260] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:40.874514 19027 ts_tablet_manager.cc:1398] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:40.874516 19036 raft_consensus.cc:2229] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.874675 19036 raft_consensus.cc:2258] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.875463 19036 ts_tablet_manager.cc:1875] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:40.875445 18396 consensus_queue.cc:1035] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.876029 19026 tablet_bootstrap.cc:492] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:40.876196 19026 ts_tablet_manager.cc:1367] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:40.876732 19026 raft_consensus.cc:348] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.876874 19026 raft_consensus.cc:374] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.876922 19026 raft_consensus.cc:729] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.877056 19026 consensus_queue.cc:260] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.877314 19026 ts_tablet_manager.cc:1398] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.877575 19026 tablet_bootstrap.cc:492] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:40.877965 19036 ts_tablet_manager.cc:1888] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:40.878079 19036 log.cc:1196] T 444c1a9d0e6943dc880161ae320673b6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/444c1a9d0e6943dc880161ae320673b6
I20240501 14:09:40.878444 19026 tablet_bootstrap.cc:654] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:40.879084 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 444c1a9d0e6943dc880161ae320673b6 (table rebalance_test_table_19 [id=6fb41dbc28114ebb92d9a108a764dfdb]) successfully deleted
I20240501 14:09:40.883132 19026 tablet_bootstrap.cc:492] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:40.883278 19026 ts_tablet_manager.cc:1367] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:40.883472 18401 consensus_queue.cc:1035] T c44df3acaa1a4f0d90f6553449d296a8 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:40.883714 19026 raft_consensus.cc:348] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.883836 19026 raft_consensus.cc:374] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:40.883891 19026 raft_consensus.cc:729] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:40.884004 19026 consensus_queue.cc:260] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.884272 19026 ts_tablet_manager.cc:1398] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:40.890345 18875 tablet.cc:1722] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:40.897032 15655 raft_consensus.cc:482] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.897145 15655 raft_consensus.cc:504] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.897560 15655 leader_election.cc:288] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:40.897794 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de192a12d14aa1a287b1473c4ea02d" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:40.897908 15221 raft_consensus.cc:2454] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:40.898160 12520 leader_election.cc:302] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:40.898327 15655 raft_consensus.cc:2781] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:40.898394 15655 raft_consensus.cc:482] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:40.898451 15655 raft_consensus.cc:3037] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.899178 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de192a12d14aa1a287b1473c4ea02d" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:40.899328 15353 raft_consensus.cc:2454] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:40.899418 15655 raft_consensus.cc:504] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.899806 15655 leader_election.cc:288] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:40.900406 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de192a12d14aa1a287b1473c4ea02d" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:40.900403 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de192a12d14aa1a287b1473c4ea02d" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:40.900519 15221 raft_consensus.cc:3037] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.900527 15353 raft_consensus.cc:3037] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:40.901340 15353 raft_consensus.cc:2454] T 28de192a12d14aa1a287b1473c4ea02d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:40.901473 15221 raft_consensus.cc:2454] T 28de192a12d14aa1a287b1473c4ea02d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:40.901988 12523 leader_election.cc:302] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:40.902158 15655 raft_consensus.cc:2781] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:40.902241 15655 raft_consensus.cc:686] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:40.902372 15655 consensus_queue.cc:237] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:40.905365 12440 catalog_manager.cc:5484] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:40.931217 15353 raft_consensus.cc:1203] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:40.943889 12849 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:40.944350 17925 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:40.963007 12586 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:40.963369 17925 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:40.966559 15221 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:40.967168 18059 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:40.988365 19047 ts_tablet_manager.cc:902] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:40.988828 19047 tablet_copy_client.cc:308] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.4:43151
I20240501 14:09:40.989456 12999 tablet_copy_service.cc:140] P a6e94562bee149c2ace25672f0f9bf50: Received BeginTabletCopySession request for tablet c2639132e2d04ce480edc800214f25be from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:45837)
I20240501 14:09:40.989590 12999 tablet_copy_service.cc:161] P a6e94562bee149c2ace25672f0f9bf50: Beginning new tablet copy session on tablet c2639132e2d04ce480edc800214f25be from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:45837: session id = 4902d2609e6d44d089590a42e70356b2-c2639132e2d04ce480edc800214f25be
I20240501 14:09:40.989950 14957 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:40.990372 15425 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:40.990672 12999 tablet_copy_source_session.cc:205] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:40.991166 12439 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:40.991294 12439 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:40.991639 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2639132e2d04ce480edc800214f25be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:40.993610 12439 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet e5d4b8858d2a43ac836505cae472f003
I20240501 14:09:40.994230 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d4b8858d2a43ac836505cae472f003 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:40.994585 19052 tablet_replica.cc:330] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:40.994773 19047 tablet_copy_client.cc:783] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:40.994773 19052 raft_consensus.cc:2229] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:40.994927 19047 tablet_copy_client.cc:647] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:40.994966 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d4b8858d2a43ac836505cae472f003 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:40.995002 19052 raft_consensus.cc:2258] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.995151 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d4b8858d2a43ac836505cae472f003 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:40 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:40.995189 18637 tablet_replica.cc:330] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:40.995343 18637 raft_consensus.cc:2229] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.995442 19053 tablet_replica.cc:330] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:40.995471 18637 raft_consensus.cc:2258] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.995558 19052 ts_tablet_manager.cc:1875] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.995587 19053 raft_consensus.cc:2229] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:40.995711 19053 raft_consensus.cc:2258] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:40.996256 19053 ts_tablet_manager.cc:1875] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.998236 19052 ts_tablet_manager.cc:1888] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:40.998369 19052 log.cc:1196] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/e5d4b8858d2a43ac836505cae472f003
I20240501 14:09:40.998785 19052 ts_tablet_manager.cc:1909] T e5d4b8858d2a43ac836505cae472f003 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:40.999197 18637 ts_tablet_manager.cc:1875] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:40.999536 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table_new_name [id=e1c76b8df8c64dbca3bd54f2b10f8047]) successfully deleted
I20240501 14:09:41.001145 18637 ts_tablet_manager.cc:1888] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.001212 18637 log.cc:1196] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/e5d4b8858d2a43ac836505cae472f003
I20240501 14:09:41.001293 19053 ts_tablet_manager.cc:1888] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.001375 19053 log.cc:1196] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/e5d4b8858d2a43ac836505cae472f003
I20240501 14:09:41.001456 18637 ts_tablet_manager.cc:1909] T e5d4b8858d2a43ac836505cae472f003 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:41.001694 19053 ts_tablet_manager.cc:1909] T e5d4b8858d2a43ac836505cae472f003 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:41.002072 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table_new_name [id=e1c76b8df8c64dbca3bd54f2b10f8047]) successfully deleted
I20240501 14:09:41.002324 15089 consensus_queue.cc:237] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:41.003286 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet e5d4b8858d2a43ac836505cae472f003 (table rebalancer_dynamic_table_new_name [id=e1c76b8df8c64dbca3bd54f2b10f8047]) successfully deleted
I20240501 14:09:41.003440 12978 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.003827 12717 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.004243 17717 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.004599 18610 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.005964 15425 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.006405 12717 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.007144 12978 raft_consensus.cc:2932] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.007720 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5957da39df64253a2b8d7f388308719 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.010353 12439 catalog_manager.cc:5484] T a5957da39df64253a2b8d7f388308719 P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 2 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.012439 19047 tablet_copy_client.cc:523] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.013046 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet a5957da39df64253a2b8d7f388308719 with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:41.013617 19054 tablet_replica.cc:330] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:41.013746 19054 raft_consensus.cc:2229] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.013836 19054 raft_consensus.cc:2258] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.014035 19047 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:41.014511 19054 ts_tablet_manager.cc:1875] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.014918 18321 raft_consensus.cc:979] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: : Instructing follower e6ab42bc38694f7f9fc8bfcc0a1483b7 to start an election
I20240501 14:09:41.015004 18321 raft_consensus.cc:1067] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Signalling peer e6ab42bc38694f7f9fc8bfcc0a1483b7 to start an election
I20240501 14:09:41.015398 12717 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc"
dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
from {username='slave'} at 127.6.82.3:56961
I20240501 14:09:41.015520 12717 raft_consensus.cc:482] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:09:41.015585 12717 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:41.016278 12717 raft_consensus.cc:504] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:41.016845 19054 ts_tablet_manager.cc:1888] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:41.016907 19054 log.cc:1196] T a5957da39df64253a2b8d7f388308719 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/a5957da39df64253a2b8d7f388308719
I20240501 14:09:41.016923 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:41.017025 12849 raft_consensus.cc:3032] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:09:41.017062 12849 raft_consensus.cc:729] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:41.017199 12849 consensus_queue.cc:260] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:41.017378 12849 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:41.018127 12849 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:41.018410 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet a5957da39df64253a2b8d7f388308719 (table rebalance_test_table_1 [id=6ba1c910e8224850ae2f01d241ffb2b5]) successfully deleted
I20240501 14:09:41.018880 19047 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.019151 19047 tablet_bootstrap.cc:492] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:09:41.019202 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:41.019192 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d9bd5e798f4bf2a5dd1eedcb4676cc" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:41.019328 14957 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:41.019328 12586 raft_consensus.cc:3037] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:41.019342 19047 ts_tablet_manager.cc:1367] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.006s sys 0.000s
I20240501 14:09:41.019814 19047 raft_consensus.cc:348] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:41.019946 19047 raft_consensus.cc:729] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:09:41.020054 19047 consensus_queue.cc:260] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:41.020277 12586 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:41.020277 14957 raft_consensus.cc:2454] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:41.020359 19047 ts_tablet_manager.cc:1398] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.020684 12653 leader_election.cc:302] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:41.020804 12717 leader_election.cc:288] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:41.020866 18656 raft_consensus.cc:2781] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:41.020937 18656 raft_consensus.cc:686] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:41.021081 18656 consensus_queue.cc:237] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:41.021492 12999 tablet_copy_service.cc:342] P a6e94562bee149c2ace25672f0f9bf50: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-c2639132e2d04ce480edc800214f25be received from {username='slave'} at 127.6.82.3:45837
I20240501 14:09:41.021580 12999 tablet_copy_service.cc:433] P a6e94562bee149c2ace25672f0f9bf50: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-c2639132e2d04ce480edc800214f25be on tablet c2639132e2d04ce480edc800214f25be with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:09:41.023520 12440 catalog_manager.cc:5484] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 1 to 2, leader changed from 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.031777 19056 ts_tablet_manager.cc:902] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:41.032276 19056 tablet_copy_client.cc:308] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.2:40113
I20240501 14:09:41.032732 12738 tablet_copy_service.cc:140] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Received BeginTabletCopySession request for tablet abac52c109174021ac11360b0d22ccfa from peer ab90e77f0027449bb3155980ca95593c ({username='slave'} at 127.6.82.6:58523)
I20240501 14:09:41.032838 12738 tablet_copy_service.cc:161] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Beginning new tablet copy session on tablet abac52c109174021ac11360b0d22ccfa from peer ab90e77f0027449bb3155980ca95593c at {username='slave'} at 127.6.82.6:58523: session id = ab90e77f0027449bb3155980ca95593c-abac52c109174021ac11360b0d22ccfa
I20240501 14:09:41.033861 12738 tablet_copy_source_session.cc:205] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.034694 19056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abac52c109174021ac11360b0d22ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.037576 19056 tablet_copy_client.cc:783] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.037825 19056 tablet_copy_client.cc:647] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.040064 19056 tablet_copy_client.cc:523] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.041411 19056 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:41.059540 12978 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:41.060003 15425 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:41.065460 12717 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:09:41.065382 15089 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:41.067013 14957 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:41.067497 17717 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:41.068637 12979 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer ab90e77f0027449bb3155980ca95593c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:41.068919 19056 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.069231 19056 tablet_bootstrap.cc:492] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: Bootstrap complete.
I20240501 14:09:41.069401 19056 ts_tablet_manager.cc:1367] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.006s
I20240501 14:09:41.069790 19056 raft_consensus.cc:348] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:41.069756 15425 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.069895 19056 raft_consensus.cc:729] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: LEARNER
I20240501 14:09:41.069975 19056 consensus_queue.cc:260] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:41.070219 19056 ts_tablet_manager.cc:1398] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.071645 12738 tablet_copy_service.cc:342] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Request end of tablet copy session ab90e77f0027449bb3155980ca95593c-abac52c109174021ac11360b0d22ccfa received from {username='slave'} at 127.6.82.6:58523
I20240501 14:09:41.071650 18610 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P ab90e77f0027449bb3155980ca95593c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:41.071725 12738 tablet_copy_service.cc:433] P e6ab42bc38694f7f9fc8bfcc0a1483b7: ending tablet copy session ab90e77f0027449bb3155980ca95593c-abac52c109174021ac11360b0d22ccfa on tablet abac52c109174021ac11360b0d22ccfa with peer ab90e77f0027449bb3155980ca95593c
I20240501 14:09:41.072546 12978 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.072638 14957 raft_consensus.cc:2932] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.072806 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68b04992f1f94b36b01dfde24a42318d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.073661 12717 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.074301 18321 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.076568 12438 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 8 to 10, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "ab90e77f0027449bb3155980ca95593c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.079248 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 68b04992f1f94b36b01dfde24a42318d with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:41.079466 19053 tablet_replica.cc:330] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:41.079592 19053 raft_consensus.cc:2229] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.079686 19053 raft_consensus.cc:2258] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.080837 19053 ts_tablet_manager.cc:1875] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.083146 19053 ts_tablet_manager.cc:1888] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:41.083199 19053 log.cc:1196] T 68b04992f1f94b36b01dfde24a42318d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/68b04992f1f94b36b01dfde24a42318d
I20240501 14:09:41.083671 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 68b04992f1f94b36b01dfde24a42318d (table rebalance_test_table_9 [id=ad525790911a41d3a5782fe26efd9e61]) successfully deleted
I20240501 14:09:41.084769 18396 raft_consensus.cc:1050] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8: attempting to promote NON_VOTER ab90e77f0027449bb3155980ca95593c to VOTER
I20240501 14:09:41.085160 18396 consensus_queue.cc:237] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:41.086393 12717 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:41.086519 15089 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:41.086853 18396 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:41.087033 19063 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:41.087102 12978 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:41.087491 19063 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:41.089303 18396 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.090272 12978 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.091112 12717 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.093279 12438 catalog_manager.cc:5484] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 2 to 3, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.094791 15089 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab90e77f0027449bb3155980ca95593c (127.6.82.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.095713 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:41.097951 15221 consensus_queue.cc:237] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:41.098479 18544 raft_consensus.cc:482] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.098592 18544 raft_consensus.cc:504] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:41.098973 18544 leader_election.cc:288] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:41.099979 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213a1449f940b7884e357f84448e27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:41.100081 15089 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.100093 15222 raft_consensus.cc:2454] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.100693 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213a1449f940b7884e357f84448e27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:41.100831 12586 raft_consensus.cc:2454] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.101413 14893 leader_election.cc:302] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:41.101644 18541 raft_consensus.cc:2781] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.101701 18541 raft_consensus.cc:482] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.101742 18541 raft_consensus.cc:3037] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.101925 12717 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.102463 19064 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.102806 19064 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.102860 18541 raft_consensus.cc:504] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:41.103209 18541 leader_election.cc:288] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:41.103549 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213a1449f940b7884e357f84448e27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:41.103677 12585 raft_consensus.cc:3037] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.104652 12585 raft_consensus.cc:2454] T 5f213a1449f940b7884e357f84448e27 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.104863 18401 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.105150 14893 leader_election.cc:302] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:41.105319 18541 raft_consensus.cc:2781] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.105255 12717 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.105404 18541 raft_consensus.cc:686] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:41.105521 18541 consensus_queue.cc:237] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:41.105963 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f213a1449f940b7884e357f84448e27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:41.106056 15221 raft_consensus.cc:3037] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.106809 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8226993467944dcb24d1efcda07538a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.106983 15221 raft_consensus.cc:2454] T 5f213a1449f940b7884e357f84448e27 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.107821 12441 catalog_manager.cc:5484] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 3 to 4, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.108319 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2c48d43dc9473db740931824cf1c21 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]), partition=
I20240501 14:09:41.108572 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2c48d43dc9473db740931824cf1c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.108709 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2c48d43dc9473db740931824cf1c21 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]), partition=
I20240501 14:09:41.108922 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2c48d43dc9473db740931824cf1c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.111047 19027 tablet_bootstrap.cc:492] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:41.111356 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2c48d43dc9473db740931824cf1c21 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]), partition=
I20240501 14:09:41.111608 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2c48d43dc9473db740931824cf1c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.111944 19027 tablet_bootstrap.cc:654] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.112612 19025 tablet_bootstrap.cc:492] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:41.113471 19025 tablet_bootstrap.cc:654] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.114094 18766 tablet_bootstrap.cc:492] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:41.114521 19047 ts_tablet_manager.cc:902] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:41.114975 19047 tablet_copy_client.cc:308] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.8:46801
I20240501 14:09:41.114989 18766 tablet_bootstrap.cc:654] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.115393 15373 tablet_copy_service.cc:140] P f23f426391724d6688aafffe7ee017d8: Received BeginTabletCopySession request for tablet 4bd8d72845814f17961145fd8d4d0bbd from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:34307)
I20240501 14:09:41.115490 15373 tablet_copy_service.cc:161] P f23f426391724d6688aafffe7ee017d8: Beginning new tablet copy session on tablet 4bd8d72845814f17961145fd8d4d0bbd from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:34307: session id = 4902d2609e6d44d089590a42e70356b2-4bd8d72845814f17961145fd8d4d0bbd
I20240501 14:09:41.115500 15089 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.116386 15373 tablet_copy_source_session.cc:205] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.117295 12438 catalog_manager.cc:5484] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.119004 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8d72845814f17961145fd8d4d0bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.119446 19025 tablet_bootstrap.cc:492] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:41.119526 19025 ts_tablet_manager.cc:1367] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:41.119755 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet e8226993467944dcb24d1efcda07538a with delete_type TABLET_DATA_TOMBSTONED (TS a6e94562bee149c2ace25672f0f9bf50 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:41.119989 19025 raft_consensus.cc:348] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.120106 19025 raft_consensus.cc:374] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.120138 19025 raft_consensus.cc:729] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.120263 19025 consensus_queue.cc:260] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.120550 19025 ts_tablet_manager.cc:1398] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.121287 19047 tablet_copy_client.cc:783] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.121428 19047 tablet_copy_client.cc:647] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.122453 18766 tablet_bootstrap.cc:492] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:41.122547 18766 ts_tablet_manager.cc:1367] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:41.123057 18766 raft_consensus.cc:348] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.123142 19080 tablet_replica.cc:330] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:41.123157 18766 raft_consensus.cc:374] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.123188 18766 raft_consensus.cc:729] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.123281 19080 raft_consensus.cc:2229] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.123296 18766 consensus_queue.cc:260] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.123422 19080 raft_consensus.cc:2258] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.123562 18766 ts_tablet_manager.cc:1398] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.125866 19080 ts_tablet_manager.cc:1875] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.126288 19027 tablet_bootstrap.cc:492] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:41.126367 19027 ts_tablet_manager.cc:1367] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240501 14:09:41.126863 19047 tablet_copy_client.cc:523] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.126817 19027 raft_consensus.cc:348] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.126924 19027 raft_consensus.cc:374] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.126955 19027 raft_consensus.cc:729] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.127094 19027 consensus_queue.cc:260] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.127408 19027 ts_tablet_manager.cc:1398] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.128250 19047 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:41.128305 19080 ts_tablet_manager.cc:1888] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:41.128393 19080 log.cc:1196] T e8226993467944dcb24d1efcda07538a P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/e8226993467944dcb24d1efcda07538a
I20240501 14:09:41.129006 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet e8226993467944dcb24d1efcda07538a (table rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]) successfully deleted
I20240501 14:09:41.132048 19047 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.132329 19047 tablet_bootstrap.cc:492] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:09:41.132529 19047 ts_tablet_manager.cc:1367] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.001s
I20240501 14:09:41.133035 19047 raft_consensus.cc:348] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:41.133152 19047 raft_consensus.cc:729] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:09:41.133276 19047 consensus_queue.cc:260] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:09:41.133641 19047 ts_tablet_manager.cc:1398] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.134106 15373 tablet_copy_service.cc:342] P f23f426391724d6688aafffe7ee017d8: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-4bd8d72845814f17961145fd8d4d0bbd received from {username='slave'} at 127.6.82.3:34307
I20240501 14:09:41.134192 15373 tablet_copy_service.cc:433] P f23f426391724d6688aafffe7ee017d8: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-4bd8d72845814f17961145fd8d4d0bbd on tablet 4bd8d72845814f17961145fd8d4d0bbd with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:09:41.141892 12717 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:41.143218 15089 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:41.143693 12849 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:09:41.144145 18287 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:41.144588 17177 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:41.146247 17177 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.146560 18541 raft_consensus.cc:482] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.146667 18541 raft_consensus.cc:504] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.146697 15089 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.147101 18541 leader_election.cc:288] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:41.147465 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee040fd7ece4b8d8af328f9205dae20" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:41.147599 15090 raft_consensus.cc:2454] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.147878 14893 leader_election.cc:302] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:41.148057 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df57dee35f924801a23bd9b5c547a55f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.148138 18541 raft_consensus.cc:2781] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.148188 18541 raft_consensus.cc:482] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.148226 18541 raft_consensus.cc:3037] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.148180 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee040fd7ece4b8d8af328f9205dae20" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:41.148312 15221 raft_consensus.cc:2454] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.149232 18541 raft_consensus.cc:504] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.149591 18541 leader_election.cc:288] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:41.149415 12849 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.151387 12443 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 6 to 8, VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) evicted. New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.151540 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee040fd7ece4b8d8af328f9205dae20" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:41.151650 15221 raft_consensus.cc:3037] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.152168 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee040fd7ece4b8d8af328f9205dae20" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:41.152271 15089 raft_consensus.cc:3037] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.152537 15221 raft_consensus.cc:2454] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.153124 14891 leader_election.cc:302] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:41.153324 18541 raft_consensus.cc:2781] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.153402 18541 raft_consensus.cc:686] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:41.153538 18541 consensus_queue.cc:237] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.155506 15089 raft_consensus.cc:2454] T cee040fd7ece4b8d8af328f9205dae20 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.155683 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet df57dee35f924801a23bd9b5c547a55f with delete_type TABLET_DATA_TOMBSTONED (TS 8a13c028ba8a4a659b74af6c265df95e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:41.156395 12443 catalog_manager.cc:5484] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.158481 19094 tablet_replica.cc:330] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:41.158686 19094 raft_consensus.cc:2229] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.158856 19094 raft_consensus.cc:2258] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.159354 19094 ts_tablet_manager.cc:1875] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.161595 19094 ts_tablet_manager.cc:1888] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240501 14:09:41.161679 19094 log.cc:1196] T df57dee35f924801a23bd9b5c547a55f P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/df57dee35f924801a23bd9b5c547a55f
I20240501 14:09:41.162235 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet df57dee35f924801a23bd9b5c547a55f (table rebalancer_extra_table_2 [id=9f7d7d9d5c834c748a44fb11031f9134]) successfully deleted
I20240501 14:09:41.177605 18541 raft_consensus.cc:482] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.177718 18541 raft_consensus.cc:504] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.178138 18541 leader_election.cc:288] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:41.178545 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2c48d43dc9473db740931824cf1c21" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:41.178714 15353 raft_consensus.cc:2454] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.178886 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2c48d43dc9473db740931824cf1c21" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c" is_pre_election: true
I20240501 14:09:41.179018 15089 raft_consensus.cc:2454] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.179065 14894 leader_election.cc:302] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:41.179234 18541 raft_consensus.cc:2781] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.179286 18541 raft_consensus.cc:482] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.179348 18541 raft_consensus.cc:3037] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.180388 18541 raft_consensus.cc:504] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.180660 15089 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:41.180680 18541 leader_election.cc:288] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:41.180970 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2c48d43dc9473db740931824cf1c21" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90e77f0027449bb3155980ca95593c"
I20240501 14:09:41.181063 15089 raft_consensus.cc:3037] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.181066 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2c48d43dc9473db740931824cf1c21" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:41.181165 15353 raft_consensus.cc:3037] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.182052 15089 raft_consensus.cc:2454] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.182053 15353 raft_consensus.cc:2454] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.182430 14893 leader_election.cc:302] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, ab90e77f0027449bb3155980ca95593c; no voters:
I20240501 14:09:41.182596 18541 raft_consensus.cc:2781] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.182554 18656 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.182663 18541 raft_consensus.cc:686] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:41.182809 18541 consensus_queue.cc:237] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:41.185511 12443 catalog_manager.cc:5484] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.193815 12585 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:41.194283 18656 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.196452 15089 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.196899 17415 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.199694 15353 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:09:41.200502 17177 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.202847 12978 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.203222 12717 consensus_queue.cc:237] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:41.203401 15878 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.204532 12586 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:41.205296 18287 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.205471 15090 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:41.206983 18286 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:41.207123 18287 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.209385 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7130a0cb5bca433d8eb682e05b7a8f29 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.210554 15090 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.211649 12585 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } }
I20240501 14:09:41.213728 12443 catalog_manager.cc:5484] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 3 to 5, VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) evicted. New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.216183 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 7130a0cb5bca433d8eb682e05b7a8f29 with delete_type TABLET_DATA_TOMBSTONED (TS f23f426391724d6688aafffe7ee017d8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:41.219228 19131 tablet_replica.cc:330] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:41.219594 19131 raft_consensus.cc:2229] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.219815 19131 raft_consensus.cc:2258] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.220471 19131 ts_tablet_manager.cc:1875] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.223023 19131 ts_tablet_manager.cc:1888] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 14:09:41.223093 19131 log.cc:1196] T 7130a0cb5bca433d8eb682e05b7a8f29 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/7130a0cb5bca433d8eb682e05b7a8f29
I20240501 14:09:41.223735 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 7130a0cb5bca433d8eb682e05b7a8f29 (table rebalancer_extra_table_10 [id=0e3bca2d16924d5caca0dad0ff9a3e40]) successfully deleted
I20240501 14:09:41.252655 12849 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.253121 18520 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.258419 12849 consensus_queue.cc:237] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:41.259855 12717 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.259974 12585 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:09:41.260113 12782 consensus_peers.cc:485] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 -> Peer 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): Couldn't send request to peer 66e2dc34330848c3b8c2dd15898d34c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c778ae9911740c2b0cb42dfbc694bb1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.260325 18321 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.260522 16502 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.262245 15878 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:41.262713 12585 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:41.262781 12717 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:41.265589 12441 catalog_manager.cc:5484] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 4, NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } } }
I20240501 14:09:41.274364 12978 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } }
I20240501 14:09:41.275789 12717 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:09:41.276204 14957 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:09:41.276350 12915 consensus_peers.cc:485] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b5c40793b348bbb0e5251c8d553e30. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:41.276458 17167 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.276763 18665 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.278439 17167 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:41.278961 12717 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:41.281677 12438 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.282305 14957 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: NON_VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: true } } }
I20240501 14:09:41.290570 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:41.295028 15353 raft_consensus.cc:1261] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.295173 15090 raft_consensus.cc:1261] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.295465 18541 consensus_queue.cc:1035] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.295661 18544 consensus_queue.cc:1035] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.298712 12438 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]). Expected version 1 got 0
I20240501 14:09:41.298719 19180 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.299245 19180 tablet.cc:1620] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:41.299288 12443 catalog_manager.cc:5576] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673) does not have the latest schema for tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]). Expected version 1 got 0
I20240501 14:09:41.300052 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]). Expected version 1 got 0
I20240501 14:09:41.300415 18874 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.300729 18874 tablet.cc:1620] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:41.302071 19181 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.302371 19181 tablet.cc:1620] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:41.302821 12438 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df]). Expected version 1 got 0
I20240501 14:09:41.303462 15655 consensus_queue.cc:1035] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.307046 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df] alter complete (version 1)
I20240501 14:09:41.307016 18238 consensus_queue.cc:1035] T 28de192a12d14aa1a287b1473c4ea02d P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.328974 12978 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.329420 18520 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.333984 12717 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.335186 18059 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.336025 15353 consensus_queue.cc:237] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:41.337167 12849 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.337426 12978 raft_consensus.cc:1261] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.337764 16475 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.338270 16475 consensus_queue.cc:1035] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.339773 16475 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.340142 12849 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.341763 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6628290de33e4aba8ccd1651d1d054dd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.342093 12978 raft_consensus.cc:2932] T 6628290de33e4aba8ccd1651d1d054dd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.343209 12439 catalog_manager.cc:5484] T 6628290de33e4aba8ccd1651d1d054dd P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.346156 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 6628290de33e4aba8ccd1651d1d054dd with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:41.346382 19053 tablet_replica.cc:330] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:41.346539 19053 raft_consensus.cc:2229] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.346657 19053 raft_consensus.cc:2258] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.347340 19053 ts_tablet_manager.cc:1875] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.349406 19053 ts_tablet_manager.cc:1888] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:41.349478 19053 log.cc:1196] T 6628290de33e4aba8ccd1651d1d054dd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/6628290de33e4aba8ccd1651d1d054dd
I20240501 14:09:41.349960 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 6628290de33e4aba8ccd1651d1d054dd (table rebalance_test_table_13 [id=baaf3b9884564526839fd42eb9fe2ccd]) successfully deleted
I20240501 14:09:41.400367 14957 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.400849 18287 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.412357 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:41.417579 12585 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.418648 19180 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:41.419126 19180 tablet.cc:1620] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:41.419081 18287 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.419181 19181 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:41.419507 19181 tablet.cc:1620] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:41.420076 18874 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:41.420338 18874 tablet.cc:1620] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: MemRowSet was empty: no flush needed.
I20240501 14:09:41.425731 12849 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 14:09:41.428037 18286 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:41.428494 12717 consensus_queue.cc:237] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:41.431402 14957 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.432152 12585 raft_consensus.cc:1261] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.432977 18286 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.433566 18286 consensus_queue.cc:1035] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.434834 18656 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.436630 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.437150 12585 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.437309 14957 raft_consensus.cc:2932] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:09:41.437824 12440 catalog_manager.cc:5484] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 5 to 7, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.440356 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:41.440618 19054 tablet_replica.cc:330] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:41.440711 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=831c58daa3ae48e3913e3150542387df] alter complete (version 2)
I20240501 14:09:41.440796 19054 raft_consensus.cc:2229] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.440923 19054 raft_consensus.cc:2258] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.441885 19054 ts_tablet_manager.cc:1875] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.444048 19054 ts_tablet_manager.cc:1888] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:41.444129 19054 log.cc:1196] T 01d9bd5e798f4bf2a5dd1eedcb4676cc P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/01d9bd5e798f4bf2a5dd1eedcb4676cc
I20240501 14:09:41.444905 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 01d9bd5e798f4bf2a5dd1eedcb4676cc (table rebalance_test_table_11 [id=d5859398307345bba93840c1752d9b15]) successfully deleted
I20240501 14:09:41.463876 12849 raft_consensus.cc:1203] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:41.469591 17167 raft_consensus.cc:1050] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER f23f426391724d6688aafffe7ee017d8 to VOTER
I20240501 14:09:41.470067 17167 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:41.471396 15353 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:41.471765 14957 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:41.471911 19273 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:41.471989 12849 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:09:41.472229 18279 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:41.472395 19266 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:09:41.475014 19266 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.475234 14957 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.477190 12849 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.478000 12441 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.479426 15353 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f23f426391724d6688aafffe7ee017d8 (127.6.82.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.486299 15353 consensus_queue.cc:237] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:41.486877 12978 consensus_queue.cc:237] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } }
I20240501 14:09:41.487624 15221 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:41.488140 14957 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:41.488461 12849 raft_consensus.cc:1261] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 14:09:41.488525 17167 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:41.489056 18520 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:41.489934 18520 consensus_queue.cc:1035] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:09:41.490404 15353 raft_consensus.cc:1261] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:09:41.490801 17167 consensus_queue.cc:1035] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:09:41.491698 18520 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.491781 19273 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.492215 14957 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.493909 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63a76e89d58d48ba9d1b3b98ef6c172d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.494885 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7693e3e204634676a758726bb6d3d2ce with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.495194 15221 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.495589 12439 catalog_manager.cc:5484] T 63a76e89d58d48ba9d1b3b98ef6c172d P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 8 to 9, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.496004 12441 catalog_manager.cc:5484] T 7693e3e204634676a758726bb6d3d2ce P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index 8 to 10, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New cstate: current_term: 2 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.497186 12849 raft_consensus.cc:2932] T 7693e3e204634676a758726bb6d3d2ce P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.499449 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 7693e3e204634676a758726bb6d3d2ce with delete_type TABLET_DATA_TOMBSTONED (TS 6a3cc0660eed427eaee38164a8c1d4c3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:41.499850 19288 tablet_replica.cc:330] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:41.501332 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a76e89d58d48ba9d1b3b98ef6c172d with delete_type TABLET_DATA_TOMBSTONED (TS 4902d2609e6d44d089590a42e70356b2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:41.502002 19054 tablet_replica.cc:330] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:41.502147 19054 raft_consensus.cc:2229] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.502333 19054 raft_consensus.cc:2258] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.503118 19054 ts_tablet_manager.cc:1875] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.505046 19054 ts_tablet_manager.cc:1888] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:09:41.505108 19054 log.cc:1196] T 63a76e89d58d48ba9d1b3b98ef6c172d P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/63a76e89d58d48ba9d1b3b98ef6c172d
I20240501 14:09:41.505559 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 63a76e89d58d48ba9d1b3b98ef6c172d (table rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]) successfully deleted
I20240501 14:09:41.506525 15353 raft_consensus.cc:2932] T 63a76e89d58d48ba9d1b3b98ef6c172d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } }
I20240501 14:09:41.500030 19288 raft_consensus.cc:2229] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.507720 19288 raft_consensus.cc:2258] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.508405 19288 ts_tablet_manager.cc:1875] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.510228 19288 ts_tablet_manager.cc:1888] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 14:09:41.510289 19288 log.cc:1196] T 7693e3e204634676a758726bb6d3d2ce P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/7693e3e204634676a758726bb6d3d2ce
I20240501 14:09:41.510802 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 7693e3e204634676a758726bb6d3d2ce (table rebalance_test_table_3 [id=dec69a1383534061bc3956918692fd47]) successfully deleted
I20240501 14:09:41.511907 12849 raft_consensus.cc:1203] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 14:09:41.512516 18541 consensus_queue.cc:1035] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.518931 18541 consensus_queue.cc:1035] T 5f213a1449f940b7884e357f84448e27 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.546277 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:41.552588 19180 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:41.553189 18874 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:41.560794 19181 tablet.cc:1722] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:41.562026 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=831c58daa3ae48e3913e3150542387df] alter complete (version 3)
I20240501 14:09:41.577939 15090 raft_consensus.cc:1203] T abac52c109174021ac11360b0d22ccfa P ab90e77f0027449bb3155980ca95593c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:09:41.591439 18544 consensus_queue.cc:1035] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.595144 18541 consensus_queue.cc:1035] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.631865 12849 consensus_queue.cc:237] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:41.633158 15090 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.633466 12978 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 14:09:41.633894 18321 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.634717 18321 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:09:41.635979 17415 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.636490 15090 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.637677 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a0e8e476bc46420d992a43bf3b4c73fd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.638525 12978 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.639292 12443 catalog_manager.cc:5484] T a0e8e476bc46420d992a43bf3b4c73fd P ab90e77f0027449bb3155980ca95593c reported cstate change: config changed from index 5 to 7, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 2 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } }
I20240501 14:09:41.642282 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8e476bc46420d992a43bf3b4c73fd with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:41.642549 19053 tablet_replica.cc:330] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:41.642689 19053 raft_consensus.cc:2229] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.642838 19053 raft_consensus.cc:2258] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.643540 19053 ts_tablet_manager.cc:1875] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:41.645660 19053 ts_tablet_manager.cc:1888] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 14:09:41.645725 19053 log.cc:1196] T a0e8e476bc46420d992a43bf3b4c73fd P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0e8e476bc46420d992a43bf3b4c73fd
I20240501 14:09:41.646278 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet a0e8e476bc46420d992a43bf3b4c73fd (table rebalance_test_table_17 [id=569c4d64cbd74ff4a20691238dabc391]) successfully deleted
I20240501 14:09:41.664871 12441 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:41.665026 12441 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:41.667104 12441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 0e2c48d43dc9473db740931824cf1c21
I20240501 14:09:41.667641 14938 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2c48d43dc9473db740931824cf1c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:41 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:41.667641 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2c48d43dc9473db740931824cf1c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:41 UTC) from {username='slave'} at 127.0.0.1:53354
I20240501 14:09:41.667919 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e2c48d43dc9473db740931824cf1c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:41 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:41.667986 19369 tablet_replica.cc:330] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: stopping tablet replica
I20240501 14:09:41.668128 19131 tablet_replica.cc:330] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:41.668260 19131 raft_consensus.cc:2229] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.668392 19131 raft_consensus.cc:2258] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.668632 19370 tablet_replica.cc:330] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:41.668794 19370 raft_consensus.cc:2229] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:41.668990 19370 raft_consensus.cc:2258] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.669011 19131 ts_tablet_manager.cc:1875] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:41.669196 19369 raft_consensus.cc:2229] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:41.669337 19369 raft_consensus.cc:2258] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:41.669503 19370 ts_tablet_manager.cc:1875] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:41.670148 19369 ts_tablet_manager.cc:1875] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:41.670241 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:41.671577 19131 ts_tablet_manager.cc:1888] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.671749 19370 ts_tablet_manager.cc:1888] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.671793 19131 log.cc:1196] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/0e2c48d43dc9473db740931824cf1c21
I20240501 14:09:41.671873 19370 log.cc:1196] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/0e2c48d43dc9473db740931824cf1c21
I20240501 14:09:41.672168 19131 ts_tablet_manager.cc:1909] T 0e2c48d43dc9473db740931824cf1c21 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:41.672243 19370 ts_tablet_manager.cc:1909] T 0e2c48d43dc9473db740931824cf1c21 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:41.673072 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table_new_name [id=831c58daa3ae48e3913e3150542387df]) successfully deleted
I20240501 14:09:41.673538 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table_new_name [id=831c58daa3ae48e3913e3150542387df]) successfully deleted
I20240501 14:09:41.673887 19369 ts_tablet_manager.cc:1888] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:41.673959 19369 log.cc:1196] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e2c48d43dc9473db740931824cf1c21
I20240501 14:09:41.674258 19369 ts_tablet_manager.cc:1909] T 0e2c48d43dc9473db740931824cf1c21 P ab90e77f0027449bb3155980ca95593c: Deleting consensus metadata
I20240501 14:09:41.675201 12431 catalog_manager.cc:4830] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): tablet 0e2c48d43dc9473db740931824cf1c21 (table rebalancer_dynamic_table_new_name [id=831c58daa3ae48e3913e3150542387df]) successfully deleted
I20240501 14:09:41.677839 19371 ts_tablet_manager.cc:902] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: Initiating tablet copy from peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:41.678356 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 0b37410addd44b5a9af05862e783590a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.678550 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 0b37410addd44b5a9af05862e783590a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.678717 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b37410addd44b5a9af05862e783590a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.678874 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b37410addd44b5a9af05862e783590a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.679289 14936 tablet_service.cc:1467] Processing CreateTablet for tablet d37260925f4a4ebdb1124097dd31ee1a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.679520 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37260925f4a4ebdb1124097dd31ee1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.681191 19374 tablet_bootstrap.cc:492] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:41.681516 15202 tablet_service.cc:1467] Processing CreateTablet for tablet d6c733eedf574248b3a03509e4c77265 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.681630 14937 tablet_service.cc:1467] Processing CreateTablet for tablet d6c733eedf574248b3a03509e4c77265 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.681795 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c733eedf574248b3a03509e4c77265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.681836 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c733eedf574248b3a03509e4c77265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.681835 15201 tablet_service.cc:1467] Processing CreateTablet for tablet d37260925f4a4ebdb1124097dd31ee1a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.682070 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37260925f4a4ebdb1124097dd31ee1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.684345 12827 tablet_service.cc:1467] Processing CreateTablet for tablet d37260925f4a4ebdb1124097dd31ee1a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.684553 15068 tablet_service.cc:1467] Processing CreateTablet for tablet 0b37410addd44b5a9af05862e783590a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.684641 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d37260925f4a4ebdb1124097dd31ee1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.684825 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b37410addd44b5a9af05862e783590a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.685014 19376 tablet_bootstrap.cc:492] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:41.685154 15331 tablet_service.cc:1467] Processing CreateTablet for tablet d6c733eedf574248b3a03509e4c77265 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=34881f8064544ba78aade2e5e2bf2413]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:41.685380 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c733eedf574248b3a03509e4c77265. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.686002 19376 tablet_bootstrap.cc:654] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.686812 19375 tablet_bootstrap.cc:492] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:41.688083 19371 tablet_copy_client.cc:308] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.3:42215
I20240501 14:09:41.688145 19375 tablet_bootstrap.cc:654] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.688527 19378 tablet_bootstrap.cc:492] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:41.688665 12869 tablet_copy_service.cc:140] P 4902d2609e6d44d089590a42e70356b2: Received BeginTabletCopySession request for tablet 7c778ae9911740c2b0cb42dfbc694bb1 from peer 66e2dc34330848c3b8c2dd15898d34c8 ({username='slave'} at 127.6.82.7:45273)
I20240501 14:09:41.688741 12869 tablet_copy_service.cc:161] P 4902d2609e6d44d089590a42e70356b2: Beginning new tablet copy session on tablet 7c778ae9911740c2b0cb42dfbc694bb1 from peer 66e2dc34330848c3b8c2dd15898d34c8 at {username='slave'} at 127.6.82.7:45273: session id = 66e2dc34330848c3b8c2dd15898d34c8-7c778ae9911740c2b0cb42dfbc694bb1
I20240501 14:09:41.689371 19378 tablet_bootstrap.cc:654] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.689698 12869 tablet_copy_source_session.cc:205] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:09:41.690181 19374 tablet_bootstrap.cc:654] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.690903 19371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c778ae9911740c2b0cb42dfbc694bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.691017 19382 tablet_bootstrap.cc:492] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c: Bootstrap starting.
I20240501 14:09:41.691593 19374 tablet_bootstrap.cc:492] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:41.691593 19375 tablet_bootstrap.cc:492] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:41.691701 19375 ts_tablet_manager.cc:1367] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:41.691751 19374 ts_tablet_manager.cc:1367] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:41.692175 19375 raft_consensus.cc:348] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.692281 19375 raft_consensus.cc:374] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.692313 19375 raft_consensus.cc:729] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.692175 19374 raft_consensus.cc:348] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.692358 19374 raft_consensus.cc:374] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.692440 19374 raft_consensus.cc:729] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.692441 19375 consensus_queue.cc:260] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.692582 19374 consensus_queue.cc:260] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.692754 19375 ts_tablet_manager.cc:1398] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.692891 19374 ts_tablet_manager.cc:1398] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.693153 19375 tablet_bootstrap.cc:492] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:41.693395 19371 tablet_copy_client.cc:783] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 0 data blocks...
I20240501 14:09:41.693542 19371 tablet_copy_client.cc:647] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:09:41.694008 19378 tablet_bootstrap.cc:492] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:41.694092 19375 tablet_bootstrap.cc:654] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.694100 19378 ts_tablet_manager.cc:1367] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:41.694622 19378 raft_consensus.cc:348] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.695201 19375 tablet_bootstrap.cc:492] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:41.695277 19378 raft_consensus.cc:374] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.695333 19375 ts_tablet_manager.cc:1367] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:41.695411 19378 raft_consensus.cc:729] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.695710 19378 consensus_queue.cc:260] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.695854 19375 raft_consensus.cc:348] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.696110 19375 raft_consensus.cc:374] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.696194 19375 raft_consensus.cc:729] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.696378 19375 consensus_queue.cc:260] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.696797 19375 ts_tablet_manager.cc:1398] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.697566 19371 tablet_copy_client.cc:523] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:09:41.699021 19371 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:41.699111 19374 tablet_bootstrap.cc:492] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:41.700006 19374 tablet_bootstrap.cc:654] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.701273 19382 tablet_bootstrap.cc:654] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.702102 19374 tablet_bootstrap.cc:492] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:41.702186 19374 ts_tablet_manager.cc:1367] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:41.702245 19378 ts_tablet_manager.cc:1398] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:41.702639 19374 raft_consensus.cc:348] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.702819 19374 raft_consensus.cc:374] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.703161 19374 raft_consensus.cc:729] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.703410 19374 consensus_queue.cc:260] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.703461 19382 tablet_bootstrap.cc:492] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c: No bootstrap required, opened a new log
I20240501 14:09:41.703547 19376 tablet_bootstrap.cc:492] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:41.703634 19376 ts_tablet_manager.cc:1367] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:41.703615 19382 ts_tablet_manager.cc:1367] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:41.703802 19374 ts_tablet_manager.cc:1398] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:41.704124 19376 raft_consensus.cc:348] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.704239 19376 raft_consensus.cc:374] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.704188 19382 raft_consensus.cc:348] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.704270 19376 raft_consensus.cc:729] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.704291 19382 raft_consensus.cc:374] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.704319 19382 raft_consensus.cc:729] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90e77f0027449bb3155980ca95593c, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.704378 19376 consensus_queue.cc:260] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.704459 19382 consensus_queue.cc:260] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.704661 19374 tablet_bootstrap.cc:492] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:41.704661 19376 ts_tablet_manager.cc:1398] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.704773 19382 ts_tablet_manager.cc:1398] T 0b37410addd44b5a9af05862e783590a P ab90e77f0027449bb3155980ca95593c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.704815 19376 tablet_bootstrap.cc:492] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:41.705839 19374 tablet_bootstrap.cc:654] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.705943 19376 tablet_bootstrap.cc:654] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.708042 19374 tablet_bootstrap.cc:492] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:41.708133 19374 ts_tablet_manager.cc:1367] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:41.709709 19376 tablet_bootstrap.cc:492] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:41.709826 19376 ts_tablet_manager.cc:1367] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:41.710284 19376 raft_consensus.cc:348] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.710393 19376 raft_consensus.cc:374] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.710434 19376 raft_consensus.cc:729] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.710579 19376 consensus_queue.cc:260] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.710785 19374 raft_consensus.cc:348] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.710886 19374 raft_consensus.cc:374] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.710917 19374 raft_consensus.cc:729] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.710935 19376 ts_tablet_manager.cc:1398] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:41.711047 19374 consensus_queue.cc:260] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.711405 19374 ts_tablet_manager.cc:1398] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:41.722965 19371 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:09:41.723230 19371 tablet_bootstrap.cc:492] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap complete.
I20240501 14:09:41.723373 19371 ts_tablet_manager.cc:1367] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.003s
I20240501 14:09:41.723788 19371 raft_consensus.cc:348] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:41.723912 19371 raft_consensus.cc:729] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: LEARNER
I20240501 14:09:41.723995 19371 consensus_queue.cc:260] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: NON_VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: true } }
I20240501 14:09:41.724243 19371 ts_tablet_manager.cc:1398] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:41.724778 12869 tablet_copy_service.cc:342] P 4902d2609e6d44d089590a42e70356b2: Request end of tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-7c778ae9911740c2b0cb42dfbc694bb1 received from {username='slave'} at 127.6.82.7:45273
I20240501 14:09:41.724879 12869 tablet_copy_service.cc:433] P 4902d2609e6d44d089590a42e70356b2: ending tablet copy session 66e2dc34330848c3b8c2dd15898d34c8-7c778ae9911740c2b0cb42dfbc694bb1 on tablet 7c778ae9911740c2b0cb42dfbc694bb1 with peer 66e2dc34330848c3b8c2dd15898d34c8
I20240501 14:09:41.744225 18396 raft_consensus.cc:482] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.744344 18396 raft_consensus.cc:504] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.744719 18396 leader_election.cc:288] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:41.745090 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c733eedf574248b3a03509e4c77265" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:41.745095 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c733eedf574248b3a03509e4c77265" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:41.745218 15353 raft_consensus.cc:2454] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:41.745232 14957 raft_consensus.cc:2454] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:41.745532 15155 leader_election.cc:302] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:41.745708 18396 raft_consensus.cc:2781] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.745766 18396 raft_consensus.cc:482] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.745821 18396 raft_consensus.cc:3037] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.746639 18396 raft_consensus.cc:504] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.746989 18396 leader_election.cc:288] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:41.747308 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c733eedf574248b3a03509e4c77265" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:41.747308 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c733eedf574248b3a03509e4c77265" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:41.747426 14957 raft_consensus.cc:3037] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.747452 15353 raft_consensus.cc:3037] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.748445 15353 raft_consensus.cc:2454] T d6c733eedf574248b3a03509e4c77265 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:41.748440 14957 raft_consensus.cc:2454] T d6c733eedf574248b3a03509e4c77265 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:41.748764 15157 leader_election.cc:302] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:41.748947 18396 raft_consensus.cc:2781] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.749033 18396 raft_consensus.cc:686] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:41.749204 18396 consensus_queue.cc:237] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.751186 12443 catalog_manager.cc:5484] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:41.768865 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:41.774137 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 63d28f6e1f9a48d5aaa54f407412ca9b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c]), partition=
I20240501 14:09:41.774206 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 63d28f6e1f9a48d5aaa54f407412ca9b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c]), partition=
I20240501 14:09:41.774420 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d28f6e1f9a48d5aaa54f407412ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.774405 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 63d28f6e1f9a48d5aaa54f407412ca9b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c]), partition=
I20240501 14:09:41.774492 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d28f6e1f9a48d5aaa54f407412ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.774645 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d28f6e1f9a48d5aaa54f407412ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:41.776721 19376 tablet_bootstrap.cc:492] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:41.776856 19375 tablet_bootstrap.cc:492] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:41.776895 19392 tablet_bootstrap.cc:492] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:41.777611 19376 tablet_bootstrap.cc:654] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.777690 19375 tablet_bootstrap.cc:654] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.777694 19392 tablet_bootstrap.cc:654] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:41.778723 19376 tablet_bootstrap.cc:492] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:41.778836 19376 ts_tablet_manager.cc:1367] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:41.779253 19392 tablet_bootstrap.cc:492] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:41.779346 19392 ts_tablet_manager.cc:1367] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:41.779310 19376 raft_consensus.cc:348] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.779415 19376 raft_consensus.cc:374] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.779451 19376 raft_consensus.cc:729] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.779584 19376 consensus_queue.cc:260] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.779862 19392 raft_consensus.cc:348] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.779991 19392 raft_consensus.cc:374] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.780038 19392 raft_consensus.cc:729] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.780171 19392 consensus_queue.cc:260] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.780462 19392 ts_tablet_manager.cc:1398] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.781224 19375 tablet_bootstrap.cc:492] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:41.781311 19375 ts_tablet_manager.cc:1367] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:41.781739 19375 raft_consensus.cc:348] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.781836 19375 raft_consensus.cc:374] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:41.781869 19375 raft_consensus.cc:729] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:41.781894 19376 ts_tablet_manager.cc:1398] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:41.781983 19375 consensus_queue.cc:260] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.782253 19375 ts_tablet_manager.cc:1398] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:41.785876 19396 ts_tablet_manager.cc:902] T 25b5c40793b348bbb0e5251c8d553e30 P ab90e77f0027449bb3155980ca95593c: Initiating tablet copy from peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
F20240501 14:09:41.786101 19396 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1ea1af845a8e4f39a973d1f8ef691762.queue_time_us, tablet-download-pool-1ea1af845a8e4f39a973d1f8ef691762.queue_time_us) (tablet-download-pool-1ea1af845a8e4f39a973d1f8ef691762.run_cpu_time_us, tablet-download-pool-1ea1af845a8e4f39a973d1f8ef691762.run_cpu_time_us) (tablet-download-pool-1ea1af845a8e4f39a973d1f8ef691762.run_wall_time_us, tablet-download-pool-1ea1af845a8e4f39a973d1f8ef691762.run_wall_time_us) (tablet-download-pool-218f8a4fae324dc7bc535e6a18addb61.queue_time_us, tablet-download-pool-218f8a4fae324dc7bc535e6a18addb61.queue_time_us) (tablet-download-pool-218f8a4fae324dc7bc535e6a18addb61.run_cpu_time_us, tablet-download-pool-218f8a4fae324dc7bc535e6a18addb61.run_cpu_time_us) (tablet-download-pool-218f8a4fae324dc7bc535e6a18addb61.run_wall_time_us, tablet-download-pool-218f8a4fae324dc7bc535e6a18addb61.run_wall_time_us) (tablet-download-pool-25b5c40793b348bbb0e5251c8d553e30.queue_time_us, tablet-download-pool-25b5c40793b348bbb0e5251c8d553e30.queue_time_us) (tablet-download-pool-515d8f8e9d744aa4b2c8a4090c2f72e8.queue_time_us, tablet-download-pool-515d8f8e9d744aa4b2c8a4090c2f72e8.queue_time_us) (tablet-download-pool-515d8f8e9d744aa4b2c8a4090c2f72e8.run_cpu_time_us, tablet-download-pool-515d8f8e9d744aa4b2c8a4090c2f72e8.run_cpu_time_us) (tablet-download-pool-515d8f8e9d744aa4b2c8a4090c2f72e8.run_wall_time_us, tablet-download-pool-515d8f8e9d744aa4b2c8a4090c2f72e8.run_wall_time_us) (tablet-download-pool-6305f3f7cb8846bf843d2d753fd1ae07.queue_time_us, tablet-download-pool-6305f3f7cb8846bf843d2d753fd1ae07.queue_time_us) (tablet-download-pool-6305f3f7cb8846bf843d2d753fd1ae07.run_cpu_time_us, tablet-download-pool-6305f3f7cb8846bf843d2d753fd1ae07.run_cpu_time_us) (tablet-download-pool-6305f3f7cb8846bf843d2d753fd1ae07.run_wall_time_us, tablet-download-pool-6305f3f7cb8846bf843d2d753fd1ae07.run_wall_time_us) (tablet-download-pool-68b04992f1f94b36b01dfde24a42318d.queue_time_us, tablet-download-pool-68b04992f1f94b36b01dfde24a42318d.queue_time_us) (tablet-download-pool-68b04992f1f94b36b01dfde24a42318d.run_cpu_time_us, tablet-download-pool-68b04992f1f94b36b01dfde24a42318d.run_cpu_time_us) (tablet-download-pool-68b04992f1f94b36b01dfde24a42318d.run_wall_time_us, tablet-download-pool-68b04992f1f94b36b01dfde24a42318d.run_wall_time_us) (tablet-download-pool-727a51852ac2428eb23f7690fec56a16.queue_time_us, tablet-download-pool-727a51852ac2428eb23f7690fec56a16.queue_time_us) (tablet-download-pool-727a51852ac2428eb23f7690fec56a16.run_cpu_time_us, tablet-download-pool-727a51852ac2428eb23f7690fec56a16.run_cpu_time_us) (tablet-download-pool-727a51852ac2428eb23f7690fec56a16.run_wall_time_us, tablet-download-pool-727a51852ac2428eb23f7690fec56a16.run_wall_time_us) (tablet-download-pool-a0e8e476bc46420d992a43bf3b4c73fd.queue_time_us, tablet-download-pool-a0e8e476bc46420d992a43bf3b4c73fd.queue_time_us) (tablet-download-pool-a0e8e476bc46420d992a43bf3b4c73fd.run_cpu_time_us, tablet-download-pool-a0e8e476bc46420d992a43bf3b4c73fd.run_cpu_time_us) (tablet-download-pool-a0e8e476bc46420d992a43bf3b4c73fd.run_wall_time_us, tablet-download-pool-a0e8e476bc46420d992a43bf3b4c73fd.run_wall_time_us) (tablet-download-pool-a5957da39df64253a2b8d7f388308719.queue_time_us, tablet-download-pool-a5957da39df64253a2b8d7f388308719.queue_time_us) (tablet-download-pool-a5957da39df64253a2b8d7f388308719.run_cpu_time_us, tablet-download-pool-a5957da39df64253a2b8d7f388308719.run_cpu_time_us) (tablet-download-pool-a5957da39df64253a2b8d7f388308719.run_wall_time_us, tablet-download-pool-a5957da39df64253a2b8d7f388308719.run_wall_time_us) (tablet-download-pool-abac52c109174021ac11360b0d22ccfa.queue_time_us, tablet-download-pool-abac52c109174021ac11360b0d22ccfa.queue_time_us) (tablet-download-pool-abac52c109174021ac11360b0d22ccfa.run_cpu_time_us, tablet-download-pool-abac52c109174021ac11360b0d22ccfa.run_cpu_time_us) (tablet-download-pool-abac52c109174021ac11360b0d22ccfa.run_wall_time_us, tablet-download-pool-abac52c109174021ac11360b0d22ccfa.run_wall_time_us) (tablet-download-pool-afa5ebe09c99435fb69ceb527ab4734a.queue_time_us, tablet-download-pool-afa5ebe09c99435fb69ceb527ab4734a.queue_time_us) (tablet-download-pool-afa5ebe09c99435fb69ceb527ab4734a.run_cpu_time_us, tablet-download-pool-afa5ebe09c99435fb69ceb527ab4734a.run_cpu_time_us) (tablet-download-pool-afa5ebe09c99435fb69ceb527ab4734a.run_wall_time_us, tablet-download-pool-afa5ebe09c99435fb69ceb527ab4734a.run_wall_time_us) (tablet-download-pool-bd5489eab5b04751bc8b3e5fb9e8cf73.queue_time_us, tablet-download-pool-bd5489eab5b04751bc8b3e5fb9e8cf73.queue_time_us) (tablet-download-pool-bd5489eab5b04751bc8b3e5fb9e8cf73.run_cpu_time_us, tablet-download-pool-bd5489eab5b04751bc8b3e5fb9e8cf73.run_cpu_time_us) (tablet-download-pool-bd5489eab5b04751bc8b3e5fb9e8cf73.run_wall_time_us, tablet-download-pool-bd5489eab5b04751bc8b3e5fb9e8cf73.run_wall_time_us) (tablet-download-pool-d2631324f66d454697b895ecdd227276.queue_time_us, tablet-download-pool-d2631324f66d454697b895ecdd227276.queue_time_us) (tablet-download-pool-d2631324f66d454697b895ecdd227276.run_cpu_time_us, tablet-download-pool-d2631324f66d454697b895ecdd227276.run_cpu_time_us) (tablet-download-pool-d2631324f66d454697b895ecdd227276.run_wall_time_us, tablet-download-pool-d2631324f66d454697b895ecdd227276.run_wall_time_us) (tablet-download-pool-e13e55ae52a949bc893de94082130157.queue_time_us, tablet-download-pool-e13e55ae52a949bc893de94082130157.queue_time_us) (tablet-download-pool-e13e55ae52a949bc893de94082130157.run_cpu_time_us, tablet-download-pool-e13e55ae52a949bc893de94082130157.run_cpu_time_us) (tablet-download-pool-e13e55ae52a949bc893de94082130157.run_wall_time_us, tablet-download-pool-e13e55ae52a949bc893de94082130157.run_wall_time_us) (tablet-download-pool-e8226993467944dcb24d1efcda07538a.queue_time_us, tablet-download-pool-e8226993467944dcb24d1efcda07538a.queue_time_us) (tablet-download-pool-e8226993467944dcb24d1efcda07538a.run_cpu_time_us, tablet-download-pool-e8226993467944dcb24d1efcda07538a.run_cpu_time_us) (tablet-download-pool-e8226993467944dcb24d1efcda07538a.run_wall_time_us, tablet-download-pool-e8226993467944dcb24d1efcda07538a.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fade3111dcd google::LogMessage::Fail() at ??:0
@ 0x7fade3115b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fade31117cc google::LogMessage::Flush() at ??:0
@ 0x7fade3112f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fade457b044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fade455ff98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fade455defc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fade711077a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fade7119a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fade719a76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fade71ae4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fade7199226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fade71a8c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d1faddb75c std::function<>::operator()() at ??:0
@ 0x7fade4562e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fade45636ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fade456503b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55d1faddb75c std::function<>::operator()() at ??:0
@ 0x7fade4552807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fade4d906db start_thread at ??:0
@ 0x7fade22c871f clone at ??:0
I20240501 14:09:41.858848 18059 raft_consensus.cc:482] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.858968 18059 raft_consensus.cc:504] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.859452 18059 leader_election.cc:288] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:41.859730 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d28f6e1f9a48d5aaa54f407412ca9b" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:41.859762 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d28f6e1f9a48d5aaa54f407412ca9b" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:41.859861 15221 raft_consensus.cc:2454] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:41.859906 12978 raft_consensus.cc:2454] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:41.860116 15286 leader_election.cc:302] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:41.860359 18059 raft_consensus.cc:2781] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.860407 18059 raft_consensus.cc:482] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.860440 18059 raft_consensus.cc:3037] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.861543 18059 raft_consensus.cc:504] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.861872 18059 leader_election.cc:288] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:41.862548 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d28f6e1f9a48d5aaa54f407412ca9b" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:41.862566 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d28f6e1f9a48d5aaa54f407412ca9b" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:41.862649 15221 raft_consensus.cc:3037] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.862684 12978 raft_consensus.cc:3037] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.863723 15221 raft_consensus.cc:2454] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:41.863852 12978 raft_consensus.cc:2454] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:41.864013 15286 leader_election.cc:302] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:41.864344 16475 raft_consensus.cc:2781] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.864435 16475 raft_consensus.cc:686] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:41.864568 16475 consensus_queue.cc:237] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:41.866660 12443 catalog_manager.cc:5484] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.926265 16475 raft_consensus.cc:1050] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8: attempting to promote NON_VOTER 4902d2609e6d44d089590a42e70356b2 to VOTER
I20240501 14:09:41.926632 16475 consensus_queue.cc:237] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:41.928026 12848 raft_consensus.cc:1261] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:41.928503 15221 raft_consensus.cc:1261] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:41.928988 18059 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:41.929577 17925 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:41.931816 16475 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.935343 15222 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.937362 12848 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.938657 12439 catalog_manager.cc:5484] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { replace: true } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.942482 15353 consensus_queue.cc:237] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:41.943240 18541 raft_consensus.cc:482] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.943347 18541 raft_consensus.cc:504] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.943710 18541 leader_election.cc:288] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:41.944025 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37260925f4a4ebdb1124097dd31ee1a" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:41.944144 15222 raft_consensus.cc:2454] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.944317 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37260925f4a4ebdb1124097dd31ee1a" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:41.944430 12848 raft_consensus.cc:2454] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.944711 14891 leader_election.cc:302] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:41.945050 18541 raft_consensus.cc:2781] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.945094 18541 raft_consensus.cc:482] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.945124 18541 raft_consensus.cc:3037] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.946132 18541 raft_consensus.cc:504] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.946483 18541 leader_election.cc:288] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:41.946764 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37260925f4a4ebdb1124097dd31ee1a" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:41.946841 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d37260925f4a4ebdb1124097dd31ee1a" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:41.946899 15222 raft_consensus.cc:3037] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.946955 12848 raft_consensus.cc:3037] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.947815 12848 raft_consensus.cc:2454] T d37260925f4a4ebdb1124097dd31ee1a P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.947815 15222 raft_consensus.cc:2454] T d37260925f4a4ebdb1124097dd31ee1a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.948103 14894 leader_election.cc:302] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:41.949174 15222 raft_consensus.cc:1261] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.949548 16475 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.951085 12848 raft_consensus.cc:1261] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:41.951828 16475 consensus_queue.cc:1035] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:41.953809 16475 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.954015 18541 raft_consensus.cc:2781] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.954113 18541 raft_consensus.cc:686] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:41.954241 18541 consensus_queue.cc:237] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:41.957937 15222 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.958931 12440 catalog_manager.cc:5484] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.959268 12848 raft_consensus.cc:2932] T 4bd8d72845814f17961145fd8d4d0bbd P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.961184 12443 catalog_manager.cc:5484] T 4bd8d72845814f17961145fd8d4d0bbd P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 3 to 4, VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:41.961699 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4bd8d72845814f17961145fd8d4d0bbd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:41.970614 18544 raft_consensus.cc:482] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:41.970723 18544 raft_consensus.cc:504] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.971097 18544 leader_election.cc:288] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:41.971436 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b37410addd44b5a9af05862e783590a" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:41.971560 15353 raft_consensus.cc:2454] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:41.971923 14894 leader_election.cc:302] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:41.972115 18544 raft_consensus.cc:2781] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:41.972167 18544 raft_consensus.cc:482] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:41.972200 18544 raft_consensus.cc:3037] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.973117 18544 raft_consensus.cc:504] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.973497 18544 leader_election.cc:288] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:41.973781 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b37410addd44b5a9af05862e783590a" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:41.973922 15353 raft_consensus.cc:3037] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:41.974905 15353 raft_consensus.cc:2454] T 0b37410addd44b5a9af05862e783590a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:41.975193 14894 leader_election.cc:302] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:41.975528 18544 raft_consensus.cc:2781] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:41.975700 18544 raft_consensus.cc:686] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:41.975903 18544 consensus_queue.cc:237] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } }
I20240501 14:09:41.978013 12439 catalog_manager.cc:5484] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:41.985023 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:41.989910 15222 raft_consensus.cc:1261] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.990092 12978 raft_consensus.cc:1261] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:41.990330 18059 consensus_queue.cc:1035] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.990486 18520 consensus_queue.cc:1035] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:41.993696 19181 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.993757 12439 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 63d28f6e1f9a48d5aaa54f407412ca9b (table rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c]). Expected version 1 got 0
I20240501 14:09:41.994040 19181 tablet.cc:1620] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:41.995826 12439 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 63d28f6e1f9a48d5aaa54f407412ca9b (table rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c]). Expected version 1 got 0
I20240501 14:09:41.997035 19481 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.997565 19481 tablet.cc:1620] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:41.997947 19482 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:41.998272 19482 tablet.cc:1620] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:42.000270 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c] alter complete (version 1)
I20240501 14:09:42.001828 12439 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 63d28f6e1f9a48d5aaa54f407412ca9b (table rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c]). Expected version 1 got 0
I20240501 14:09:42.043982 17167 raft_consensus.cc:1050] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50: attempting to promote NON_VOTER 4902d2609e6d44d089590a42e70356b2 to VOTER
I20240501 14:09:42.044401 17167 consensus_queue.cc:237] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:42.045966 15222 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:42.046509 14957 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:42.046582 12848 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 14:09:42.047358 17167 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:42.047828 17167 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:42.048969 17167 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:09:42.050354 19483 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.050769 14957 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.053032 12848 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.054754 12439 catalog_manager.cc:5484] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 2 to 3, 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:42.057729 12978 consensus_queue.cc:237] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:42.059233 14957 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.059593 12848 raft_consensus.cc:1261] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:09:42.059957 19483 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.060406 19485 consensus_queue.cc:1035] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:09:42.062135 17167 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.062287 12848 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.063760 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2639132e2d04ce480edc800214f25be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:42.064997 14957 raft_consensus.cc:2932] T c2639132e2d04ce480edc800214f25be P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.065413 12443 catalog_manager.cc:5484] T c2639132e2d04ce480edc800214f25be P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 3 to 4, VOTER 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } }
I20240501 14:09:42.069097 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet c2639132e2d04ce480edc800214f25be with delete_type TABLET_DATA_TOMBSTONED (TS 66e2dc34330848c3b8c2dd15898d34c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:42.069653 19491 tablet_replica.cc:330] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:42.069782 19491 raft_consensus.cc:2229] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.069890 19491 pending_rounds.cc:62] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Trying to abort 1 pending ops.
I20240501 14:09:42.069960 19491 pending_rounds.cc:68] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022889296052936704 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c2639132e2d04ce480edc800214f25be" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } } }
I20240501 14:09:42.070109 19491 raft_consensus.cc:2894] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 14:09:42.070168 19491 raft_consensus.cc:2866] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:09:42.070222 19491 raft_consensus.cc:2258] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.070802 19491 ts_tablet_manager.cc:1875] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:42.072870 19491 ts_tablet_manager.cc:1888] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 14:09:42.072943 19491 log.cc:1196] T c2639132e2d04ce480edc800214f25be P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/c2639132e2d04ce480edc800214f25be
I20240501 14:09:42.073447 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet c2639132e2d04ce480edc800214f25be (table rebalancer_extra_table_12 [id=55928b84097e43df9d2dd57a7d04a789]) successfully deleted
I20240501 14:09:42.102768 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240501 14:09:42.106288 14893 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.106343 12653 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.106456 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.106456 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.106591 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.82.6:37673 (error 108)
W20240501 14:09:42.107300 12784 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.107435 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:42.109673 19181 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.110046 19181 tablet.cc:1620] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
W20240501 14:09:42.110821 12522 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.110944 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.111166 19497 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:42.109143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.82.6:37673 (local address 127.6.82.3:35051)
0501 14:09:42.110482 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:42.110544 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":61,"threads_started":1}
W20240501 14:09:42.111261 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.111274 15156 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.111313 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.111352 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.111371 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.111407 19498 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:42.109330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.82.6:37673 (local address 127.6.82.5:39009)
0501 14:09:42.109673 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:42.109687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:42.109778 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:42.110278 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":54,"threads_started":1}
W20240501 14:09:42.111534 14893 leader_election.cc:334] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.111676 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.111696 12431 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.111732 14893 leader_election.cc:334] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.111757 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.111804 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.113355 15288 connection.cc:535] client connection to 127.6.82.6:37673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.113482 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:09:42.113852 19482 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.114152 19482 tablet.cc:1620] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:42.115013 19481 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.115326 19481 tablet.cc:1620] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
W20240501 14:09:42.117983 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.119105 19496 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:09:42.108492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.82.6:37673 (local address 127.6.82.4:43165)
0501 14:09:42.108783 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0501 14:09:42.108799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 14:09:42.108888 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:09:42.118359 (+ 9471us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":115,"threads_started":1}
I20240501 14:09:42.119235 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=f7ea16de4269409dbf45c08d765b7d8c] alter complete (version 2)
W20240501 14:09:42.119278 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:09:42.119731 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.120404 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.120733 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.120716 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:42.120802 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.126245 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.129976 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.188709 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.211527 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.220547 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:42.225466 19181 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.226168 19481 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.228143 19482 tablet.cc:1722] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.231107 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=f7ea16de4269409dbf45c08d765b7d8c] alter complete (version 3)
W20240501 14:09:42.233969 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.244660 15222 raft_consensus.cc:1203] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 14:09:42.251240 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:42.257196 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.282589 18396 consensus_queue.cc:1035] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.285701 19552 consensus_queue.cc:1035] T d6c733eedf574248b3a03509e4c77265 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.334331 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:42.334499 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:42.336592 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 63d28f6e1f9a48d5aaa54f407412ca9b
I20240501 14:09:42.337268 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d28f6e1f9a48d5aaa54f407412ca9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:42.337283 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d28f6e1f9a48d5aaa54f407412ca9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:42.337325 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d28f6e1f9a48d5aaa54f407412ca9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:42.337492 19491 tablet_replica.cc:330] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:42.337630 19556 tablet_replica.cc:330] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:42.337630 19555 tablet_replica.cc:330] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:42.337642 19491 raft_consensus.cc:2229] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.337764 19491 raft_consensus.cc:2258] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.337769 19555 raft_consensus.cc:2229] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:42.337769 19556 raft_consensus.cc:2229] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.337881 19556 raft_consensus.cc:2258] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.337987 19555 raft_consensus.cc:2258] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.338497 19556 ts_tablet_manager.cc:1875] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.338572 19491 ts_tablet_manager.cc:1875] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.338683 19555 ts_tablet_manager.cc:1875] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.340839 19491 ts_tablet_manager.cc:1888] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.340890 19555 ts_tablet_manager.cc:1888] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.340920 19491 log.cc:1196] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/63d28f6e1f9a48d5aaa54f407412ca9b
I20240501 14:09:42.341014 19555 log.cc:1196] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/63d28f6e1f9a48d5aaa54f407412ca9b
I20240501 14:09:42.341090 19556 ts_tablet_manager.cc:1888] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.341200 19491 ts_tablet_manager.cc:1909] T 63d28f6e1f9a48d5aaa54f407412ca9b P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:42.341168 19556 log.cc:1196] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/63d28f6e1f9a48d5aaa54f407412ca9b
I20240501 14:09:42.341295 19555 ts_tablet_manager.cc:1909] T 63d28f6e1f9a48d5aaa54f407412ca9b P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:42.341763 19556 ts_tablet_manager.cc:1909] T 63d28f6e1f9a48d5aaa54f407412ca9b P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:42.342100 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 63d28f6e1f9a48d5aaa54f407412ca9b (table rebalancer_dynamic_table_new_name [id=f7ea16de4269409dbf45c08d765b7d8c]) successfully deleted
I20240501 14:09:42.342126 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 63d28f6e1f9a48d5aaa54f407412ca9b (table rebalancer_dynamic_table_new_name [id=f7ea16de4269409dbf45c08d765b7d8c]) successfully deleted
I20240501 14:09:42.342514 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 63d28f6e1f9a48d5aaa54f407412ca9b (table rebalancer_dynamic_table_new_name [id=f7ea16de4269409dbf45c08d765b7d8c]) successfully deleted
W20240501 14:09:42.380436 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:42.438935 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:42.443791 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 0653fc9292d14bb69931e3feeffd7beb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be]), partition=
I20240501 14:09:42.443845 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 0653fc9292d14bb69931e3feeffd7beb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be]), partition=
I20240501 14:09:42.444090 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0653fc9292d14bb69931e3feeffd7beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.444108 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0653fc9292d14bb69931e3feeffd7beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.446373 19572 tablet_bootstrap.cc:492] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:42.446615 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 0653fc9292d14bb69931e3feeffd7beb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be]), partition=
I20240501 14:09:42.446920 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0653fc9292d14bb69931e3feeffd7beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.447032 19573 tablet_bootstrap.cc:492] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:42.447605 19572 tablet_bootstrap.cc:654] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.447861 19573 tablet_bootstrap.cc:654] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.449194 19576 tablet_bootstrap.cc:492] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:42.449092 19573 tablet_bootstrap.cc:492] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:42.449301 19573 ts_tablet_manager.cc:1367] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:42.449620 19572 tablet_bootstrap.cc:492] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:42.449718 19572 ts_tablet_manager.cc:1367] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:42.449779 19573 raft_consensus.cc:348] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.449903 19573 raft_consensus.cc:374] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.449944 19573 raft_consensus.cc:729] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.450085 19573 consensus_queue.cc:260] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.450170 19576 tablet_bootstrap.cc:654] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.450134 19572 raft_consensus.cc:348] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.450269 19572 raft_consensus.cc:374] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.450312 19572 raft_consensus.cc:729] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.450387 19573 ts_tablet_manager.cc:1398] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.450455 19572 consensus_queue.cc:260] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.450795 19572 ts_tablet_manager.cc:1398] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.451381 19576 tablet_bootstrap.cc:492] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:42.451475 19576 ts_tablet_manager.cc:1367] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:42.451869 19576 raft_consensus.cc:348] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.451969 19576 raft_consensus.cc:374] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.452010 19576 raft_consensus.cc:729] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.452137 19576 consensus_queue.cc:260] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.452410 19576 ts_tablet_manager.cc:1398] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:42.472335 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:42.482939 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:42.489588 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 71f866d9595d46aa8dcdd684727a2748 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.489599 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 7da1741653524e6e87ee6a1f8862b789 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.489650 12827 tablet_service.cc:1467] Processing CreateTablet for tablet fcae79571f1845e0b63a065180019d47 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.489962 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae79571f1845e0b63a065180019d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.489966 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f866d9595d46aa8dcdd684727a2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.490152 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da1741653524e6e87ee6a1f8862b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.490504 12698 tablet_service.cc:1467] Processing CreateTablet for tablet fcae79571f1845e0b63a065180019d47 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.490873 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae79571f1845e0b63a065180019d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.492295 19580 tablet_bootstrap.cc:492] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:42.493458 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 71f866d9595d46aa8dcdd684727a2748 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.493513 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 7da1741653524e6e87ee6a1f8862b789 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.493513 15200 tablet_service.cc:1467] Processing CreateTablet for tablet fcae79571f1845e0b63a065180019d47 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.493641 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 7da1741653524e6e87ee6a1f8862b789 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.493721 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f866d9595d46aa8dcdd684727a2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.493873 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae79571f1845e0b63a065180019d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.493892 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da1741653524e6e87ee6a1f8862b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.494045 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da1741653524e6e87ee6a1f8862b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.494118 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 71f866d9595d46aa8dcdd684727a2748 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=4bc0c22db50b40fc93c7d3232ca30257]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:42.494311 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71f866d9595d46aa8dcdd684727a2748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:42.496049 19576 tablet_bootstrap.cc:492] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:42.496206 19572 tablet_bootstrap.cc:492] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:42.496836 19576 tablet_bootstrap.cc:654] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.497112 19572 tablet_bootstrap.cc:654] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.497789 19573 tablet_bootstrap.cc:492] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:42.498476 19580 tablet_bootstrap.cc:654] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.498893 19573 tablet_bootstrap.cc:654] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.500061 19572 tablet_bootstrap.cc:492] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:42.500432 18826 consensus_queue.cc:1035] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.500190 19572 ts_tablet_manager.cc:1367] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:42.501458 19572 raft_consensus.cc:348] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.501674 19573 tablet_bootstrap.cc:492] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:42.501808 19573 ts_tablet_manager.cc:1367] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:42.502439 19576 tablet_bootstrap.cc:492] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:42.502537 19576 ts_tablet_manager.cc:1367] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:09:42.502557 19572 raft_consensus.cc:374] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.502686 19572 raft_consensus.cc:729] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.502948 19572 consensus_queue.cc:260] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.503027 19576 raft_consensus.cc:348] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.503139 19576 raft_consensus.cc:374] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.503183 19576 raft_consensus.cc:729] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.503286 19576 consensus_queue.cc:260] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.503587 19576 ts_tablet_manager.cc:1398] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:42.503820 19576 tablet_bootstrap.cc:492] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:42.504501 19572 ts_tablet_manager.cc:1398] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:42.504657 19572 tablet_bootstrap.cc:492] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:42.505231 19581 tablet_bootstrap.cc:492] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:42.506115 19576 tablet_bootstrap.cc:654] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.502393 19573 raft_consensus.cc:348] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.506438 19573 raft_consensus.cc:374] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.506538 19573 raft_consensus.cc:729] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.506764 19573 consensus_queue.cc:260] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.507802 19573 ts_tablet_manager.cc:1398] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 14:09:42.507974 19573 tablet_bootstrap.cc:492] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:42.507984 19580 tablet_bootstrap.cc:492] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:42.508066 19580 ts_tablet_manager.cc:1367] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 14:09:42.508435 19580 raft_consensus.cc:348] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.508524 19580 raft_consensus.cc:374] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.508551 19580 raft_consensus.cc:729] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.508656 19580 consensus_queue.cc:260] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.508944 19580 ts_tablet_manager.cc:1398] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:42.509574 19573 tablet_bootstrap.cc:654] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.509949 19572 tablet_bootstrap.cc:654] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.510712 18541 consensus_queue.cc:1035] T d37260925f4a4ebdb1124097dd31ee1a P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.510802 18520 raft_consensus.cc:482] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.510876 18520 raft_consensus.cc:504] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.511219 18520 leader_election.cc:288] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:42.511487 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0653fc9292d14bb69931e3feeffd7beb" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:42.511585 14957 raft_consensus.cc:2454] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:42.511554 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0653fc9292d14bb69931e3feeffd7beb" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:42.511660 19572 tablet_bootstrap.cc:492] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:42.511662 15221 raft_consensus.cc:2454] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:42.511727 19572 ts_tablet_manager.cc:1367] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 14:09:42.511883 15287 leader_election.cc:302] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:42.512104 18520 raft_consensus.cc:2781] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.512154 18520 raft_consensus.cc:482] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.512132 19572 raft_consensus.cc:348] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.512188 18520 raft_consensus.cc:3037] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.512233 19572 raft_consensus.cc:374] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.512269 19572 raft_consensus.cc:729] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.512375 19572 consensus_queue.cc:260] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.512800 19573 tablet_bootstrap.cc:492] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:42.512871 19573 ts_tablet_manager.cc:1367] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:09:42.513221 18520 raft_consensus.cc:504] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.513324 19573 raft_consensus.cc:348] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.513396 19573 raft_consensus.cc:374] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.513432 19573 raft_consensus.cc:729] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.513515 18520 leader_election.cc:288] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:42.513525 19573 consensus_queue.cc:260] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.513846 19573 ts_tablet_manager.cc:1398] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:42.514235 19572 ts_tablet_manager.cc:1398] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:42.514326 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0653fc9292d14bb69931e3feeffd7beb" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:42.514423 15222 raft_consensus.cc:3037] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.515400 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0653fc9292d14bb69931e3feeffd7beb" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:42.515502 14957 raft_consensus.cc:3037] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.515499 15222 raft_consensus.cc:2454] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:42.515856 15286 leader_election.cc:302] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:42.516022 18520 raft_consensus.cc:2781] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.516083 18520 raft_consensus.cc:686] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:42.516184 18520 consensus_queue.cc:237] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:42.516568 14957 raft_consensus.cc:2454] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:42.518452 19581 tablet_bootstrap.cc:654] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.519773 19581 tablet_bootstrap.cc:492] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:42.519879 19581 ts_tablet_manager.cc:1367] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240501 14:09:42.520071 19576 tablet_bootstrap.cc:492] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:42.520145 19576 ts_tablet_manager.cc:1367] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240501 14:09:42.520126 12443 catalog_manager.cc:5484] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.520505 19581 raft_consensus.cc:348] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.520603 19581 raft_consensus.cc:374] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.520632 19581 raft_consensus.cc:729] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.520588 19576 raft_consensus.cc:348] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.520690 19576 raft_consensus.cc:374] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.520721 19576 raft_consensus.cc:729] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.520783 19581 consensus_queue.cc:260] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.520815 19576 consensus_queue.cc:260] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.521070 19576 ts_tablet_manager.cc:1398] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:42.521070 19581 ts_tablet_manager.cc:1398] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:42.521217 19576 tablet_bootstrap.cc:492] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:42.522002 19576 tablet_bootstrap.cc:654] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:42.524047 19576 tablet_bootstrap.cc:492] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:42.524816 19576 ts_tablet_manager.cc:1367] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:42.525239 19576 raft_consensus.cc:348] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.525321 19576 raft_consensus.cc:374] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:42.525352 19576 raft_consensus.cc:729] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:42.525466 19576 consensus_queue.cc:260] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.525715 19576 ts_tablet_manager.cc:1398] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:42.539175 19552 raft_consensus.cc:482] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.539271 19552 raft_consensus.cc:504] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.539692 19552 leader_election.cc:288] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:42.539994 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da1741653524e6e87ee6a1f8862b789" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:42.540024 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da1741653524e6e87ee6a1f8862b789" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:42.540124 14957 raft_consensus.cc:2454] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:42.540124 15353 raft_consensus.cc:2454] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:42.540357 15157 leader_election.cc:302] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:42.540534 19552 raft_consensus.cc:2781] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.540596 19552 raft_consensus.cc:482] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.540645 19552 raft_consensus.cc:3037] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.541498 19552 raft_consensus.cc:504] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.541796 19552 leader_election.cc:288] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:42.542088 18826 consensus_queue.cc:1035] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.542244 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da1741653524e6e87ee6a1f8862b789" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:42.542119 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da1741653524e6e87ee6a1f8862b789" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:42.542344 15353 raft_consensus.cc:3037] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.542388 14957 raft_consensus.cc:3037] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.543324 15353 raft_consensus.cc:2454] T 7da1741653524e6e87ee6a1f8862b789 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:42.543452 14957 raft_consensus.cc:2454] T 7da1741653524e6e87ee6a1f8862b789 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:42.543565 15157 leader_election.cc:302] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:42.543721 19552 raft_consensus.cc:2781] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.543807 19552 raft_consensus.cc:686] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:42.543967 19552 consensus_queue.cc:237] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:42.546170 12441 catalog_manager.cc:5484] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.576707 18396 raft_consensus.cc:482] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.576831 18396 raft_consensus.cc:504] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.577239 18396 leader_election.cc:288] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:42.577508 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f866d9595d46aa8dcdd684727a2748" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:42.577517 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f866d9595d46aa8dcdd684727a2748" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:42.577626 14957 raft_consensus.cc:2454] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:42.577636 15351 raft_consensus.cc:2454] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:42.577838 15155 leader_election.cc:302] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:42.577989 18396 raft_consensus.cc:2781] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.578042 18396 raft_consensus.cc:482] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.578086 18396 raft_consensus.cc:3037] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.578939 18396 raft_consensus.cc:504] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.579249 18396 leader_election.cc:288] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:42.579545 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f866d9595d46aa8dcdd684727a2748" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:42.579541 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71f866d9595d46aa8dcdd684727a2748" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:42.579643 14957 raft_consensus.cc:3037] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.579651 15351 raft_consensus.cc:3037] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.580356 15351 raft_consensus.cc:2454] T 71f866d9595d46aa8dcdd684727a2748 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:42.580569 14957 raft_consensus.cc:2454] T 71f866d9595d46aa8dcdd684727a2748 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:42.580838 15155 leader_election.cc:302] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:42.581240 18396 raft_consensus.cc:2781] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.581327 18396 raft_consensus.cc:686] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:42.581444 18396 consensus_queue.cc:237] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:42.583371 12439 catalog_manager.cc:5484] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.628176 18656 raft_consensus.cc:482] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:42.628294 18656 raft_consensus.cc:504] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.628720 18656 leader_election.cc:288] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:42.629171 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae79571f1845e0b63a065180019d47" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:42.629315 15222 raft_consensus.cc:2454] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:42.629539 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae79571f1845e0b63a065180019d47" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:42.629664 12848 raft_consensus.cc:2454] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:42.630396 12651 leader_election.cc:302] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:42.630551 18656 raft_consensus.cc:2781] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:42.630587 18656 raft_consensus.cc:482] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:42.630618 18656 raft_consensus.cc:3037] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.631654 18656 raft_consensus.cc:504] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.631991 18656 leader_election.cc:288] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:42.632283 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae79571f1845e0b63a065180019d47" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:42.632395 15222 raft_consensus.cc:3037] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.633338 15222 raft_consensus.cc:2454] T fcae79571f1845e0b63a065180019d47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:42.633553 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcae79571f1845e0b63a065180019d47" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:42.633699 12848 raft_consensus.cc:3037] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:42.634750 12848 raft_consensus.cc:2454] T fcae79571f1845e0b63a065180019d47 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:42.635063 12654 leader_election.cc:302] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:42.635215 18656 raft_consensus.cc:2781] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:42.635294 18656 raft_consensus.cc:686] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:42.635345 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:42.635419 18656 consensus_queue.cc:237] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:42.637537 12441 catalog_manager.cc:5484] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:42.640929 15222 raft_consensus.cc:1261] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:42.641302 16475 consensus_queue.cc:1035] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.641499 14957 raft_consensus.cc:1261] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:42.641850 16475 consensus_queue.cc:1035] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:42.644727 12441 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 0653fc9292d14bb69931e3feeffd7beb (table rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be]). Expected version 1 got 0
I20240501 14:09:42.645483 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 0653fc9292d14bb69931e3feeffd7beb (table rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be]). Expected version 1 got 0
I20240501 14:09:42.646759 19181 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:42.647087 19181 tablet.cc:1620] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:42.649004 19482 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:42.649284 19482 tablet.cc:1620] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:42.650605 12441 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 0653fc9292d14bb69931e3feeffd7beb (table rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be]). Expected version 1 got 0
I20240501 14:09:42.652192 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be] alter complete (version 1)
I20240501 14:09:42.652341 19508 raft_consensus.cc:1050] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2: attempting to promote NON_VOTER 66e2dc34330848c3b8c2dd15898d34c8 to VOTER
I20240501 14:09:42.652771 19508 consensus_queue.cc:237] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:42.654721 12585 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:42.654855 15222 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEARNER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:42.654867 19618 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:42.655171 19618 tablet.cc:1620] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:42.655380 12717 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:09:42.655633 17415 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:42.655792 19549 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:42.656445 19549 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:09:42.659085 17415 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.659535 15222 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.661649 12717 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.661944 12443 catalog_manager.cc:5484] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:42.662844 12585 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.666214 12848 consensus_queue.cc:237] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } }
I20240501 14:09:42.667321 15222 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:42.667419 12585 raft_consensus.cc:1261] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:09:42.667788 19620 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:42.668002 17415 consensus_queue.cc:1035] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:09:42.670442 19549 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.671000 12585 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.671967 12432 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c778ae9911740c2b0cb42dfbc694bb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:09:42.673132 15222 raft_consensus.cc:2932] T 7c778ae9911740c2b0cb42dfbc694bb1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } } }
I20240501 14:09:42.673223 12438 catalog_manager.cc:5484] T 7c778ae9911740c2b0cb42dfbc694bb1 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 5 to 6, VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) evicted. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:42.675454 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c778ae9911740c2b0cb42dfbc694bb1 with delete_type TABLET_DATA_TOMBSTONED (TS e6ab42bc38694f7f9fc8bfcc0a1483b7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:42.675774 19628 tablet_replica.cc:330] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:42.675907 19628 raft_consensus.cc:2229] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.676059 19628 raft_consensus.cc:2258] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.676579 19628 ts_tablet_manager.cc:1875] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:09:42.678206 19628 ts_tablet_manager.cc:1888] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:09:42.678272 19628 log.cc:1196] T 7c778ae9911740c2b0cb42dfbc694bb1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c778ae9911740c2b0cb42dfbc694bb1
I20240501 14:09:42.678790 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 7c778ae9911740c2b0cb42dfbc694bb1 (table rebalance_test_table_5 [id=bbc0a1569d2c43caa2e15ec8f84e86b8]) successfully deleted
I20240501 14:09:42.756263 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:42.761533 19181 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.761878 19181 tablet.cc:1620] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:42.761967 19482 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.762138 19618 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:42.762225 19482 tablet.cc:1620] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:42.762411 19618 tablet.cc:1620] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:42.767125 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=ccce3e4651ef498989b06ef1683ab0be] alter complete (version 2)
I20240501 14:09:42.870285 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:42.876237 19181 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.876754 19618 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.880782 19482 tablet.cc:1722] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:42.882160 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=ccce3e4651ef498989b06ef1683ab0be] alter complete (version 3)
W20240501 14:09:42.919971 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:42.934777 18826 raft_consensus.cc:482] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:42.934913 18826 raft_consensus.cc:504] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:42.935384 18826 leader_election.cc:288] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:42.935756 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
W20240501 14:09:42.936367 14893 leader_election.cc:334] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:42.936441 14893 leader_election.cc:302] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e; no voters: ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7
I20240501 14:09:42.936559 18826 raft_consensus.cc:2726] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:42.961138 19584 raft_consensus.cc:482] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:42.961275 19584 raft_consensus.cc:504] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:42.961670 19584 leader_election.cc:288] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:42.962064 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:42.962206 14957 raft_consensus.cc:2454] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:42.962473 12652 leader_election.cc:302] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:42.962648 19584 raft_consensus.cc:2781] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:42.962709 19584 raft_consensus.cc:482] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:42.962777 19584 raft_consensus.cc:3037] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:42.962862 12653 leader_election.cc:334] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:42.963899 19584 raft_consensus.cc:504] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:42.964244 19584 leader_election.cc:288] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:42.964571 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4010a078895473980e7108269ea7bc1" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:42.964664 14957 raft_consensus.cc:3037] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:42.965112 12653 leader_election.cc:334] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:42.965394 14957 raft_consensus.cc:2454] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:42.965651 12652 leader_election.cc:302] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:42.965806 19584 raft_consensus.cc:2781] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:42.965888 19584 raft_consensus.cc:686] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:42.966033 19584 consensus_queue.cc:237] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } }
I20240501 14:09:42.968078 12441 catalog_manager.cc:5484] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 1 to 2, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:42.984542 12441 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:42.984674 12441 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:42.986447 12441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 0653fc9292d14bb69931e3feeffd7beb
I20240501 14:09:42.987044 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 0653fc9292d14bb69931e3feeffd7beb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:42.987062 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 0653fc9292d14bb69931e3feeffd7beb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:42.987095 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet 0653fc9292d14bb69931e3feeffd7beb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:42 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:42.987372 19659 tablet_replica.cc:330] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:42.987404 19658 tablet_replica.cc:330] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:42.987519 19660 tablet_replica.cc:330] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:42.987522 19658 raft_consensus.cc:2229] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:42.987524 19659 raft_consensus.cc:2229] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.987630 19659 raft_consensus.cc:2258] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.987658 19660 raft_consensus.cc:2229] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:42.987715 19658 raft_consensus.cc:2258] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.987816 19660 raft_consensus.cc:2258] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:42.988209 19659 ts_tablet_manager.cc:1875] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.988265 19658 ts_tablet_manager.cc:1875] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.988376 19660 ts_tablet_manager.cc:1875] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:42.990514 19660 ts_tablet_manager.cc:1888] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.990516 19659 ts_tablet_manager.cc:1888] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.990526 19658 ts_tablet_manager.cc:1888] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:42.990581 19660 log.cc:1196] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/0653fc9292d14bb69931e3feeffd7beb
I20240501 14:09:42.990585 19658 log.cc:1196] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/0653fc9292d14bb69931e3feeffd7beb
I20240501 14:09:42.990585 19659 log.cc:1196] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/0653fc9292d14bb69931e3feeffd7beb
I20240501 14:09:42.990914 19659 ts_tablet_manager.cc:1909] T 0653fc9292d14bb69931e3feeffd7beb P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:42.990938 19660 ts_tablet_manager.cc:1909] T 0653fc9292d14bb69931e3feeffd7beb P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:42.990968 19658 ts_tablet_manager.cc:1909] T 0653fc9292d14bb69931e3feeffd7beb P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:42.991650 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 0653fc9292d14bb69931e3feeffd7beb (table rebalancer_dynamic_table_new_name [id=ccce3e4651ef498989b06ef1683ab0be]) successfully deleted
I20240501 14:09:42.991650 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 0653fc9292d14bb69931e3feeffd7beb (table rebalancer_dynamic_table_new_name [id=ccce3e4651ef498989b06ef1683ab0be]) successfully deleted
I20240501 14:09:42.991744 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 0653fc9292d14bb69931e3feeffd7beb (table rebalancer_dynamic_table_new_name [id=ccce3e4651ef498989b06ef1683ab0be]) successfully deleted
I20240501 14:09:42.997408 18520 raft_consensus.cc:482] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:42.997507 18520 raft_consensus.cc:504] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:42.997874 18520 leader_election.cc:288] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:42.998687 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:42.998988 18826 raft_consensus.cc:482] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
W20240501 14:09:42.999101 15288 leader_election.cc:334] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:42.999087 18826 raft_consensus.cc:504] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:42.999275 15288 leader_election.cc:302] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f23f426391724d6688aafffe7ee017d8; no voters: a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c
I20240501 14:09:42.999415 18520 raft_consensus.cc:2726] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:42.999454 18826 leader_election.cc:288] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:42.999908 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
W20240501 14:09:43.000753 14893 leader_election.cc:334] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.000815 14893 leader_election.cc:302] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e; no voters: a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.000944 18826 raft_consensus.cc:2726] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:43.017143 19552 raft_consensus.cc:482] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.017287 19552 raft_consensus.cc:504] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:43.017670 19552 leader_election.cc:288] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:43.018286 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
W20240501 14:09:43.018626 15156 leader_election.cc:334] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.018844 15156 leader_election.cc:302] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8; no voters: 6a3cc0660eed427eaee38164a8c1d4c3, ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.019003 19552 raft_consensus.cc:2726] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:43.024724 19584 raft_consensus.cc:482] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.024827 19584 raft_consensus.cc:504] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:43.025202 19584 leader_election.cc:288] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:43.025619 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
W20240501 14:09:43.026482 12653 leader_election.cc:334] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.026540 12653 leader_election.cc:302] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: a6e94562bee149c2ace25672f0f9bf50, ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.026675 19584 raft_consensus.cc:2726] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:09:43.038884 19578 raft_consensus.cc:482] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.039048 19578 raft_consensus.cc:504] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.039454 19578 leader_election.cc:288] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:43.039881 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
W20240501 14:09:43.042721 12522 leader_election.cc:334] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.042817 12522 leader_election.cc:302] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3; no voters: ab90e77f0027449bb3155980ca95593c, e6ab42bc38694f7f9fc8bfcc0a1483b7
I20240501 14:09:43.042961 19578 raft_consensus.cc:2726] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:43.061825 19552 consensus_queue.cc:1035] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.062135 19549 raft_consensus.cc:482] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.062243 19549 raft_consensus.cc:504] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:43.062633 19549 leader_election.cc:288] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.063601 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:43.065063 19551 consensus_queue.cc:1035] T 7da1741653524e6e87ee6a1f8862b789 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:43.065665 12784 leader_election.cc:334] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.069192 12785 leader_election.cc:302] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2; no voters: ab90e77f0027449bb3155980ca95593c, f23f426391724d6688aafffe7ee017d8
I20240501 14:09:43.069491 19549 raft_consensus.cc:2726] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:09:43.088203 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:43.093009 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 6d381cfd9ae54c03a044567453d2d91c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe7f851cb6c94d51906f2cf913be848b]), partition=
I20240501 14:09:43.093161 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 6d381cfd9ae54c03a044567453d2d91c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe7f851cb6c94d51906f2cf913be848b]), partition=
I20240501 14:09:43.093289 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d381cfd9ae54c03a044567453d2d91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.093359 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d381cfd9ae54c03a044567453d2d91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.095875 19664 tablet_bootstrap.cc:492] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:43.097689 19663 tablet_bootstrap.cc:492] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:43.098829 19664 tablet_bootstrap.cc:654] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.099009 19663 tablet_bootstrap.cc:654] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.100029 19664 tablet_bootstrap.cc:492] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:43.100315 19663 tablet_bootstrap.cc:492] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:43.100399 19663 ts_tablet_manager.cc:1367] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:43.100128 19664 ts_tablet_manager.cc:1367] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:43.100881 19663 raft_consensus.cc:348] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.101318 19663 raft_consensus.cc:374] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.101383 19663 raft_consensus.cc:729] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.101537 19663 consensus_queue.cc:260] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.101790 19664 raft_consensus.cc:348] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.101941 19664 raft_consensus.cc:374] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.101989 19663 ts_tablet_manager.cc:1398] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.102025 19664 raft_consensus.cc:729] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.102177 19664 consensus_queue.cc:260] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.102459 19664 ts_tablet_manager.cc:1398] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.107740 19490 raft_consensus.cc:482] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.107867 19490 raft_consensus.cc:504] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:43.108253 19490 leader_election.cc:288] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:43.108734 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:43.108880 12717 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 2.
I20240501 14:09:43.109153 12913 leader_election.cc:302] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
W20240501 14:09:43.109297 12915 leader_election.cc:334] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.109325 19490 raft_consensus.cc:2781] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:09:43.109375 19490 raft_consensus.cc:482] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.109411 19490 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:43.110589 19490 raft_consensus.cc:504] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
I20240501 14:09:43.110949 19490 leader_election.cc:288] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:43.111230 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5957da39df64253a2b8d7f388308719" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:43.111349 12717 raft_consensus.cc:3037] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:09:43.112005 12915 leader_election.cc:334] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.112349 12717 raft_consensus.cc:2454] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 3.
I20240501 14:09:43.112529 19490 raft_consensus.cc:482] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.112607 19490 raft_consensus.cc:504] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:43.112628 12913 leader_election.cc:302] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.112808 19624 raft_consensus.cc:2781] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:09:43.112921 19490 leader_election.cc:288] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.112967 19624 raft_consensus.cc:686] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [term 3 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:43.113102 19624 consensus_queue.cc:237] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } }
W20240501 14:09:43.114267 12915 leader_election.cc:334] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.114265 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:43.114370 15351 raft_consensus.cc:2454] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:43.114615 12916 leader_election.cc:302] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.114823 19624 raft_consensus.cc:2781] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:43.114887 19624 raft_consensus.cc:482] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.114941 19624 raft_consensus.cc:3037] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.115078 12441 catalog_manager.cc:5484] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 2 to 3, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 3 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.115761 19624 raft_consensus.cc:504] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:43.116104 19624 leader_election.cc:288] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 2 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
W20240501 14:09:43.117213 12915 leader_election.cc:334] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.117417 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d521005dcbf74e4fb9a67fe9992ce540" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:43.117539 15351 raft_consensus.cc:3037] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.118278 15351 raft_consensus.cc:2454] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 2.
I20240501 14:09:43.118551 12916 leader_election.cc:302] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.118711 19624 raft_consensus.cc:2781] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:43.118829 19624 raft_consensus.cc:686] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [term 2 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:43.118956 19624 consensus_queue.cc:237] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } }
I20240501 14:09:43.120672 12441 catalog_manager.cc:5484] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 1 to 2, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 2 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:09:43.136283 19551 consensus_queue.cc:1035] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.137941 18520 raft_consensus.cc:482] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.138049 18520 raft_consensus.cc:504] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:43.138392 18520 leader_election.cc:288] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
W20240501 14:09:43.140308 15288 leader_election.cc:334] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.140450 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:43.140583 12848 raft_consensus.cc:2454] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:43.140806 19552 consensus_queue.cc:1035] T 71f866d9595d46aa8dcdd684727a2748 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.141034 15289 leader_election.cc:302] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.141229 18520 raft_consensus.cc:2781] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:43.141280 18520 raft_consensus.cc:482] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.141306 18520 raft_consensus.cc:3037] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.142230 18520 raft_consensus.cc:504] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:43.142586 18520 leader_election.cc:288] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:43.143187 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6a7ed55367493e9edd0461d6da6313" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:43.143294 12848 raft_consensus.cc:3037] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:09:43.143548 15288 leader_election.cc:334] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.144137 12848 raft_consensus.cc:2454] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 2.
I20240501 14:09:43.144418 15289 leader_election.cc:302] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.144585 18520 raft_consensus.cc:2781] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:43.144656 18520 raft_consensus.cc:686] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [term 2 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:43.144764 18520 consensus_queue.cc:237] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } }
I20240501 14:09:43.146538 12438 catalog_manager.cc:5484] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 1 to 2, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 2 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.149839 19584 raft_consensus.cc:482] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.149943 19584 raft_consensus.cc:504] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.150221 19584 leader_election.cc:288] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:43.150822 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:43.150977 12585 raft_consensus.cc:2454] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
W20240501 14:09:43.151273 12653 leader_election.cc:334] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.151422 12653 leader_election.cc:302] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.151582 19584 raft_consensus.cc:2781] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:43.151648 19584 raft_consensus.cc:482] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.151690 19584 raft_consensus.cc:3037] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.152282 18656 consensus_queue.cc:1035] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.152685 19584 raft_consensus.cc:504] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.153005 19584 leader_election.cc:288] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
W20240501 14:09:43.153259 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): CreateTablet RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.153657 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e8de8cd4d2407c99ea4c5520d007f5" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:43.153788 12585 raft_consensus.cc:3037] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.154841 12585 raft_consensus.cc:2454] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 2.
I20240501 14:09:43.155599 12653 leader_election.cc:302] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
W20240501 14:09:43.155740 12653 leader_election.cc:334] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.155954 18656 raft_consensus.cc:2781] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:43.156114 18656 raft_consensus.cc:686] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:43.156248 18656 consensus_queue.cc:237] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.157227 19684 consensus_queue.cc:1035] T fcae79571f1845e0b63a065180019d47 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.158377 12438 catalog_manager.cc:5484] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 1 to 2, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:43.160495 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:43.164629 19624 raft_consensus.cc:482] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.164739 19624 raft_consensus.cc:504] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:43.165151 19624 leader_election.cc:288] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:43.165541 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:43.165685 14957 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 2.
I20240501 14:09:43.165995 12914 leader_election.cc:302] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:43.166191 19624 raft_consensus.cc:2781] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:09:43.166246 19624 raft_consensus.cc:482] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.166287 19624 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:43.166771 15202 tablet_service.cc:1467] Processing CreateTablet for tablet e4fa2a1485714a1a8a8ad26c066ba36d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.167057 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fa2a1485714a1a8a8ad26c066ba36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.167466 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 57a50211285c4672969e17f460bff9c5 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.167654 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a50211285c4672969e17f460bff9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.167614 19624 raft_consensus.cc:504] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:43.167908 19624 leader_election.cc:288] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:43.168268 15331 tablet_service.cc:1467] Processing CreateTablet for tablet e4fa2a1485714a1a8a8ad26c066ba36d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.168537 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fa2a1485714a1a8a8ad26c066ba36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.169322 19690 tablet_bootstrap.cc:492] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:43.169358 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 57a50211285c4672969e17f460bff9c5 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.169379 14936 tablet_service.cc:1467] Processing CreateTablet for tablet e4fa2a1485714a1a8a8ad26c066ba36d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.169590 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a50211285c4672969e17f460bff9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.169654 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4fa2a1485714a1a8a8ad26c066ba36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.170281 19690 tablet_bootstrap.cc:654] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.170580 19663 tablet_bootstrap.cc:492] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:43.171555 19663 tablet_bootstrap.cc:654] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.171552 19664 tablet_bootstrap.cc:492] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:43.171751 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 57a50211285c4672969e17f460bff9c5 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.171993 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a50211285c4672969e17f460bff9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.172116 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68b04992f1f94b36b01dfde24a42318d" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:43.172214 14957 raft_consensus.cc:3037] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:09:43.172443 19664 tablet_bootstrap.cc:654] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.173162 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 93e0d8d2276845828598234cad411c3d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.173199 14957 raft_consensus.cc:2454] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 3.
I20240501 14:09:43.173308 19663 tablet_bootstrap.cc:492] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:43.173349 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0d8d2276845828598234cad411c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.173382 19663 ts_tablet_manager.cc:1367] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:43.173928 19663 raft_consensus.cc:348] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.174052 19663 raft_consensus.cc:374] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.174088 19663 raft_consensus.cc:729] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.174201 19663 consensus_queue.cc:260] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.174553 19663 ts_tablet_manager.cc:1398] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.174721 19663 tablet_bootstrap.cc:492] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
W20240501 14:09:43.175125 12915 leader_election.cc:334] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.175621 19663 tablet_bootstrap.cc:654] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.176865 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 93e0d8d2276845828598234cad411c3d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.177130 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0d8d2276845828598234cad411c3d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:09:43.178050 12915 leader_election.cc:334] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.178359 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 93e0d8d2276845828598234cad411c3d (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=894469d137e0467fb1f16ddc68c4f849]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.178622 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e0d8d2276845828598234cad411c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.178673 19691 tablet_bootstrap.cc:492] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:43.179841 12914 leader_election.cc:302] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.180023 19624 raft_consensus.cc:2781] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:09:43.180112 19624 raft_consensus.cc:686] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [term 3 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:43.180249 19624 consensus_queue.cc:237] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:09:43.181574 19664 tablet_bootstrap.cc:492] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:43.181654 19664 ts_tablet_manager.cc:1367] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:43.181890 19691 tablet_bootstrap.cc:654] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.182077 19664 raft_consensus.cc:348] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.182176 19664 raft_consensus.cc:374] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.182205 19664 raft_consensus.cc:729] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.182339 19664 consensus_queue.cc:260] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.182394 12438 catalog_manager.cc:5484] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 2 to 3, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 3 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.182612 19664 ts_tablet_manager.cc:1398] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.183210 19663 tablet_bootstrap.cc:492] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:43.183300 19663 ts_tablet_manager.cc:1367] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240501 14:09:43.183514 19664 tablet_bootstrap.cc:492] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:43.183804 19663 raft_consensus.cc:348] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.183916 19663 raft_consensus.cc:374] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.183950 19663 raft_consensus.cc:729] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.184087 19663 consensus_queue.cc:260] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.184387 19663 ts_tablet_manager.cc:1398] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:43.184410 19664 tablet_bootstrap.cc:654] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.184594 19663 tablet_bootstrap.cc:492] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:43.185590 19663 tablet_bootstrap.cc:654] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.186849 19663 tablet_bootstrap.cc:492] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:43.186863 19690 tablet_bootstrap.cc:492] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:43.186960 19690 ts_tablet_manager.cc:1367] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:43.187026 19663 ts_tablet_manager.cc:1367] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:43.187194 19664 tablet_bootstrap.cc:492] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:43.187263 19664 ts_tablet_manager.cc:1367] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:43.187431 19690 raft_consensus.cc:348] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.187544 19690 raft_consensus.cc:374] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.187574 19690 raft_consensus.cc:729] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.187662 19664 raft_consensus.cc:348] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.187695 19690 consensus_queue.cc:260] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.187739 19664 raft_consensus.cc:374] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.187768 19664 raft_consensus.cc:729] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.187898 19664 consensus_queue.cc:260] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.188205 19663 raft_consensus.cc:348] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.188313 19663 raft_consensus.cc:374] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.188347 19663 raft_consensus.cc:729] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.188390 19664 ts_tablet_manager.cc:1398] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:43.188477 19663 consensus_queue.cc:260] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.188777 19663 ts_tablet_manager.cc:1398] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.188896 19690 ts_tablet_manager.cc:1398] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.189071 19690 tablet_bootstrap.cc:492] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:43.192083 19690 tablet_bootstrap.cc:654] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.192931 19696 tablet_bootstrap.cc:492] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:43.193706 19696 tablet_bootstrap.cc:654] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.194660 19690 tablet_bootstrap.cc:492] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:43.194679 19696 tablet_bootstrap.cc:492] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:43.194773 19690 ts_tablet_manager.cc:1367] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:43.194787 19696 ts_tablet_manager.cc:1367] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:43.195191 19696 raft_consensus.cc:348] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.195297 19696 raft_consensus.cc:374] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.195240 19690 raft_consensus.cc:348] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.195328 19696 raft_consensus.cc:729] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.195348 19690 raft_consensus.cc:374] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.195384 19690 raft_consensus.cc:729] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.195487 19690 consensus_queue.cc:260] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.195744 19690 ts_tablet_manager.cc:1398] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:43.195451 19696 consensus_queue.cc:260] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.196573 19696 ts_tablet_manager.cc:1398] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.197412 19691 tablet_bootstrap.cc:492] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:43.197507 19691 ts_tablet_manager.cc:1367] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:43.197961 19691 raft_consensus.cc:348] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.198068 19691 raft_consensus.cc:374] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.198107 19691 raft_consensus.cc:729] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.198225 19691 consensus_queue.cc:260] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.198494 19691 ts_tablet_manager.cc:1398] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:43.213124 18544 raft_consensus.cc:482] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.213240 18544 raft_consensus.cc:504] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.213621 18544 leader_election.cc:288] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.214067 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d381cfd9ae54c03a044567453d2d91c" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:43.214219 15351 raft_consensus.cc:2454] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:43.214509 14894 leader_election.cc:302] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
W20240501 14:09:43.214813 14893 leader_election.cc:334] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.214849 18544 raft_consensus.cc:2781] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.214891 18544 raft_consensus.cc:482] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.214932 18544 raft_consensus.cc:3037] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.215747 18544 raft_consensus.cc:504] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.216017 18544 leader_election.cc:288] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.216364 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d381cfd9ae54c03a044567453d2d91c" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:43.216511 15351 raft_consensus.cc:3037] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:43.216898 14893 leader_election.cc:334] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.217403 15351 raft_consensus.cc:2454] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:43.217649 14894 leader_election.cc:302] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.217788 18544 raft_consensus.cc:2781] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.217877 18544 raft_consensus.cc:686] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:43.218037 18544 consensus_queue.cc:237] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.220185 12443 catalog_manager.cc:5484] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:43.253046 19578 raft_consensus.cc:482] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.253165 19578 raft_consensus.cc:504] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:43.253584 19578 leader_election.cc:288] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:43.254029 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:43.254179 15222 raft_consensus.cc:2454] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:43.254504 12520 leader_election.cc:302] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
W20240501 14:09:43.254642 12522 leader_election.cc:334] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.254662 19578 raft_consensus.cc:2781] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:09:43.254823 19578 raft_consensus.cc:482] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab90e77f0027449bb3155980ca95593c)
I20240501 14:09:43.254886 19578 raft_consensus.cc:3037] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.255844 19578 raft_consensus.cc:504] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:43.256175 19578 leader_election.cc:288] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:43.256536 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b037b6095a644fda854cf0365b1a7df" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:43.256639 15222 raft_consensus.cc:3037] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:09:43.257591 15222 raft_consensus.cc:2454] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 2.
I20240501 14:09:43.258034 12520 leader_election.cc:302] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:43.258229 19578 raft_consensus.cc:2781] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:09:43.258358 19578 raft_consensus.cc:686] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
W20240501 14:09:43.258425 12522 leader_election.cc:334] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.258500 19578 consensus_queue.cc:237] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:43.260491 12443 catalog_manager.cc:5484] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 1 to 2, leader changed from ab90e77f0027449bb3155980ca95593c (127.6.82.6) to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 2 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:43.274305 19624 raft_consensus.cc:482] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.274444 19624 raft_consensus.cc:504] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.274910 19624 leader_election.cc:288] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.275246 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a50211285c4672969e17f460bff9c5" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:43.275303 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a50211285c4672969e17f460bff9c5" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:43.275378 15222 raft_consensus.cc:2454] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:43.275436 15351 raft_consensus.cc:2454] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:43.275662 12913 leader_election.cc:302] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:43.275807 19624 raft_consensus.cc:2781] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.275862 19624 raft_consensus.cc:482] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.275911 19624 raft_consensus.cc:3037] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.276855 19624 raft_consensus.cc:504] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.277196 19624 leader_election.cc:288] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.277474 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a50211285c4672969e17f460bff9c5" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:43.277578 15222 raft_consensus.cc:3037] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.277853 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a50211285c4672969e17f460bff9c5" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:43.277953 15351 raft_consensus.cc:3037] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.278646 15222 raft_consensus.cc:2454] T 57a50211285c4672969e17f460bff9c5 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:43.279037 15351 raft_consensus.cc:2454] T 57a50211285c4672969e17f460bff9c5 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:43.279042 12913 leader_election.cc:302] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:43.279201 19624 raft_consensus.cc:2781] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.279310 19624 raft_consensus.cc:686] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:43.279441 19624 consensus_queue.cc:237] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:43.281378 12443 catalog_manager.cc:5484] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:43.320405 18541 raft_consensus.cc:482] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.320524 18541 raft_consensus.cc:504] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.321344 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa2a1485714a1a8a8ad26c066ba36d" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:43.321384 18541 leader_election.cc:288] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.321482 15222 raft_consensus.cc:2454] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:43.321758 14891 leader_election.cc:302] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:43.321913 18544 raft_consensus.cc:2781] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.321965 18544 raft_consensus.cc:482] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.322011 18544 raft_consensus.cc:3037] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.322029 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa2a1485714a1a8a8ad26c066ba36d" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:43.322175 15351 raft_consensus.cc:2454] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:43.323101 18544 raft_consensus.cc:504] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.323423 18544 leader_election.cc:288] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:43.323832 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa2a1485714a1a8a8ad26c066ba36d" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:43.323968 15221 raft_consensus.cc:3037] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.324928 15221 raft_consensus.cc:2454] T e4fa2a1485714a1a8a8ad26c066ba36d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:43.325158 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4fa2a1485714a1a8a8ad26c066ba36d" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:43.325187 14891 leader_election.cc:302] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:43.325273 15351 raft_consensus.cc:3037] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.325276 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:43.325345 18544 raft_consensus.cc:2781] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.325409 18544 raft_consensus.cc:686] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:43.325546 18544 consensus_queue.cc:237] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.326221 15351 raft_consensus.cc:2454] T e4fa2a1485714a1a8a8ad26c066ba36d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:43.327674 12438 catalog_manager.cc:5484] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:43.330014 15351 raft_consensus.cc:1261] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:43.331065 18544 consensus_queue.cc:1035] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:43.331193 14893 consensus_peers.cc:485] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.334439 19618 tablet.cc:1722] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.334779 19618 tablet.cc:1620] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:43.335254 12443 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 6d381cfd9ae54c03a044567453d2d91c (table rebalancer_dynamic_table [id=fe7f851cb6c94d51906f2cf913be848b]). Expected version 1 got 0
I20240501 14:09:43.336189 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 6d381cfd9ae54c03a044567453d2d91c (table rebalancer_dynamic_table [id=fe7f851cb6c94d51906f2cf913be848b]). Expected version 1 got 0
I20240501 14:09:43.336633 19181 tablet.cc:1722] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.336938 19181 tablet.cc:1620] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:43.340691 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=fe7f851cb6c94d51906f2cf913be848b] alter complete (version 1)
I20240501 14:09:43.393620 18059 raft_consensus.cc:482] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.393738 18059 raft_consensus.cc:504] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.394140 18059 leader_election.cc:288] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:43.394516 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0d8d2276845828598234cad411c3d" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:43.394560 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0d8d2276845828598234cad411c3d" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:43.394634 12848 raft_consensus.cc:2454] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:43.394699 14957 raft_consensus.cc:2454] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:43.394942 15289 leader_election.cc:302] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:43.395118 18059 raft_consensus.cc:2781] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.395185 18059 raft_consensus.cc:482] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.395233 18059 raft_consensus.cc:3037] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.395998 18059 raft_consensus.cc:504] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.396363 18059 leader_election.cc:288] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:43.396621 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0d8d2276845828598234cad411c3d" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:43.396729 12848 raft_consensus.cc:3037] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.396693 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e0d8d2276845828598234cad411c3d" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:43.396819 14957 raft_consensus.cc:3037] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.397711 12848 raft_consensus.cc:2454] T 93e0d8d2276845828598234cad411c3d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:43.397784 14957 raft_consensus.cc:2454] T 93e0d8d2276845828598234cad411c3d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:43.398072 15287 leader_election.cc:302] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:43.398213 18059 raft_consensus.cc:2781] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.398366 18059 raft_consensus.cc:686] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:43.398519 18059 consensus_queue.cc:237] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.400928 12443 catalog_manager.cc:5484] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:43.414656 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.444952 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:43.449746 19618 tablet.cc:1722] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:43.450079 19618 tablet.cc:1620] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:43.450227 19181 tablet.cc:1722] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:43.450520 19181 tablet.cc:1620] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:43.454623 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=fe7f851cb6c94d51906f2cf913be848b] alter complete (version 2)
I20240501 14:09:43.548043 14957 raft_consensus.cc:1261] T f4010a078895473980e7108269ea7bc1 P 8a13c028ba8a4a659b74af6c265df95e [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:43.548411 19684 consensus_queue.cc:1035] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.552913 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.556783 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:43.561625 19618 tablet.cc:1722] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:43.562206 19181 tablet.cc:1722] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:43.567101 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=fe7f851cb6c94d51906f2cf913be848b] alter complete (version 3)
I20240501 14:09:43.631529 15351 raft_consensus.cc:1261] T d521005dcbf74e4fb9a67fe9992ce540 P f23f426391724d6688aafffe7ee017d8 [term 2 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:43.632122 19624 consensus_queue.cc:1035] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.635928 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.658512 12848 raft_consensus.cc:1261] T 3d6a7ed55367493e9edd0461d6da6313 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 14:09:43.658999 18059 consensus_queue.cc:1035] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:09:43.662679 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.663378 12717 raft_consensus.cc:1261] T a5957da39df64253a2b8d7f388308719 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 3 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 14:09:43.663779 19490 consensus_queue.cc:1035] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:09:43.665719 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:43.668206 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.669631 14957 raft_consensus.cc:1261] T 68b04992f1f94b36b01dfde24a42318d P 8a13c028ba8a4a659b74af6c265df95e [term 3 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 14:09:43.670025 12441 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:43.670156 12441 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:43.670235 19736 consensus_queue.cc:1035] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:09:43.672323 12441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 6d381cfd9ae54c03a044567453d2d91c
I20240501 14:09:43.673624 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d381cfd9ae54c03a044567453d2d91c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:43.673890 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d381cfd9ae54c03a044567453d2d91c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:43 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:43.674300 19738 tablet_replica.cc:330] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
W20240501 14:09:43.674382 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.674619 19738 raft_consensus.cc:2229] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:43.674855 19738 raft_consensus.cc:2258] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.675436 19738 ts_tablet_manager.cc:1875] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.675493 19739 tablet_replica.cc:330] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:43.675671 19739 raft_consensus.cc:2229] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:43.675817 19739 raft_consensus.cc:2258] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:43.676362 19739 ts_tablet_manager.cc:1875] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:43.677634 19738 ts_tablet_manager.cc:1888] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.677695 19738 log.cc:1196] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/6d381cfd9ae54c03a044567453d2d91c
I20240501 14:09:43.678086 19739 ts_tablet_manager.cc:1888] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:43.678153 19739 log.cc:1196] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/6d381cfd9ae54c03a044567453d2d91c
I20240501 14:09:43.678309 19738 ts_tablet_manager.cc:1909] T 6d381cfd9ae54c03a044567453d2d91c P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:43.678427 19739 ts_tablet_manager.cc:1909] T 6d381cfd9ae54c03a044567453d2d91c P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:43.679055 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 6d381cfd9ae54c03a044567453d2d91c (table rebalancer_dynamic_table_new_name [id=fe7f851cb6c94d51906f2cf913be848b]) successfully deleted
W20240501 14:09:43.679257 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.679489 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 6d381cfd9ae54c03a044567453d2d91c (table rebalancer_dynamic_table_new_name [id=fe7f851cb6c94d51906f2cf913be848b]) successfully deleted
I20240501 14:09:43.709458 15221 raft_consensus.cc:1261] T 5b037b6095a644fda854cf0365b1a7df P 66e2dc34330848c3b8c2dd15898d34c8 [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:09:43.709890 19578 consensus_queue.cc:1035] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:43.718915 12585 raft_consensus.cc:1261] T 79e8de8cd4d2407c99ea4c5520d007f5 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:09:43.719338 19584 consensus_queue.cc:1035] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:09:43.733973 18544 consensus_queue.cc:1035] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.736974 18544 consensus_queue.cc:1035] T e4fa2a1485714a1a8a8ad26c066ba36d P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.744154 19624 consensus_queue.cc:1035] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.748925 19490 consensus_queue.cc:1035] T 57a50211285c4672969e17f460bff9c5 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.775487 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:43.781368 14936 tablet_service.cc:1467] Processing CreateTablet for tablet ae21044b255848aab31cf7d3094cbe2f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ce2423da7134e9d8e2e2a9c3288f968]), partition=
I20240501 14:09:43.781668 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae21044b255848aab31cf7d3094cbe2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.783687 19758 tablet_bootstrap.cc:492] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:43.783864 12698 tablet_service.cc:1467] Processing CreateTablet for tablet ae21044b255848aab31cf7d3094cbe2f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ce2423da7134e9d8e2e2a9c3288f968]), partition=
I20240501 14:09:43.784123 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae21044b255848aab31cf7d3094cbe2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.784874 19758 tablet_bootstrap.cc:654] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.785928 19758 tablet_bootstrap.cc:492] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:43.786011 19758 ts_tablet_manager.cc:1367] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:43.786314 19760 tablet_bootstrap.cc:492] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:43.786443 19758 raft_consensus.cc:348] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.786557 19758 raft_consensus.cc:374] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.786589 19758 raft_consensus.cc:729] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.786794 19758 consensus_queue.cc:260] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.787807 19760 tablet_bootstrap.cc:654] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.787860 19758 ts_tablet_manager.cc:1398] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.788790 19760 tablet_bootstrap.cc:492] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:43.788888 19760 ts_tablet_manager.cc:1367] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:43.789330 19760 raft_consensus.cc:348] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.789431 19760 raft_consensus.cc:374] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.789523 19760 raft_consensus.cc:729] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.789690 19760 consensus_queue.cc:260] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.790019 19760 ts_tablet_manager.cc:1398] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.824934 18059 consensus_queue.cc:1035] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.829447 18520 consensus_queue.cc:1035] T 93e0d8d2276845828598234cad411c3d P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:43.849179 19684 raft_consensus.cc:482] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.849299 19684 raft_consensus.cc:504] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.849764 19684 leader_election.cc:288] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:43.850160 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae21044b255848aab31cf7d3094cbe2f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:43.850308 14957 raft_consensus.cc:2454] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:43.850627 12652 leader_election.cc:302] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
W20240501 14:09:43.850796 12653 leader_election.cc:334] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.850816 19684 raft_consensus.cc:2781] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.850885 19684 raft_consensus.cc:482] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.850937 19684 raft_consensus.cc:3037] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.851768 19684 raft_consensus.cc:504] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.852087 19684 leader_election.cc:288] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:43.852484 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae21044b255848aab31cf7d3094cbe2f" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:43.852608 14957 raft_consensus.cc:3037] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:43.853008 12653 leader_election.cc:334] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.853515 14957 raft_consensus.cc:2454] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:43.853778 12652 leader_election.cc:302] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.853947 19684 raft_consensus.cc:2781] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.854055 19684 raft_consensus.cc:686] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:43.854200 19684 consensus_queue.cc:237] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:43.856407 12443 catalog_manager.cc:5484] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:43.900019 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.917011 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:43.922721 15331 tablet_service.cc:1467] Processing CreateTablet for tablet c0bf116648874fc3a3ff54c7addf5e2d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.923123 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bf116648874fc3a3ff54c7addf5e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.923238 15202 tablet_service.cc:1467] Processing CreateTablet for tablet c0bf116648874fc3a3ff54c7addf5e2d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.923274 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6e2acdbdc747a8a863d6e46d0d1b41 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.923552 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bf116648874fc3a3ff54c7addf5e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.923643 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6e2acdbdc747a8a863d6e46d0d1b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.925321 19766 tablet_bootstrap.cc:492] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:43.926157 19767 tablet_bootstrap.cc:492] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:43.926311 19766 tablet_bootstrap.cc:654] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.926245 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6e2acdbdc747a8a863d6e46d0d1b41 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.926271 14937 tablet_service.cc:1467] Processing CreateTablet for tablet ff37bd5ee4274e1eb2f7e461aa3b57c1 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.926524 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6e2acdbdc747a8a863d6e46d0d1b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.926985 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff37bd5ee4274e1eb2f7e461aa3b57c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.927114 19767 tablet_bootstrap.cc:654] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.928303 12827 tablet_service.cc:1467] Processing CreateTablet for tablet ff37bd5ee4274e1eb2f7e461aa3b57c1 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.928537 15200 tablet_service.cc:1467] Processing CreateTablet for tablet ff37bd5ee4274e1eb2f7e461aa3b57c1 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=797295b675324ed38972c94c50f3ffc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:43.928604 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff37bd5ee4274e1eb2f7e461aa3b57c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.928699 19758 tablet_bootstrap.cc:492] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:43.928754 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff37bd5ee4274e1eb2f7e461aa3b57c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:43.929560 19758 tablet_bootstrap.cc:654] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.930783 19758 tablet_bootstrap.cc:492] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:43.930863 19758 ts_tablet_manager.cc:1367] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:43.931169 19766 tablet_bootstrap.cc:492] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:43.931258 19766 ts_tablet_manager.cc:1367] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:43.931314 19758 raft_consensus.cc:348] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.931434 19758 raft_consensus.cc:374] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.931471 19758 raft_consensus.cc:729] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.931583 19758 consensus_queue.cc:260] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.931643 19767 tablet_bootstrap.cc:492] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:43.931725 19767 ts_tablet_manager.cc:1367] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:43.931777 19766 raft_consensus.cc:348] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.931864 19766 raft_consensus.cc:374] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.931882 19758 ts_tablet_manager.cc:1398] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.931895 19766 raft_consensus.cc:729] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.932013 19766 consensus_queue.cc:260] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.932248 19758 tablet_bootstrap.cc:492] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:43.932235 19767 raft_consensus.cc:348] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.932310 19766 ts_tablet_manager.cc:1398] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.932343 19767 raft_consensus.cc:374] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.932372 19767 raft_consensus.cc:729] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.932466 19766 tablet_bootstrap.cc:492] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:43.932551 19767 consensus_queue.cc:260] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.933200 19771 tablet_bootstrap.cc:492] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:43.933300 19766 tablet_bootstrap.cc:654] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.933825 19758 tablet_bootstrap.cc:654] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.934540 19771 tablet_bootstrap.cc:654] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.935191 19758 tablet_bootstrap.cc:492] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:43.935257 19758 ts_tablet_manager.cc:1367] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:43.935643 19771 tablet_bootstrap.cc:492] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:43.935742 19771 ts_tablet_manager.cc:1367] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:43.935681 19758 raft_consensus.cc:348] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.935789 19758 raft_consensus.cc:374] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.935819 19758 raft_consensus.cc:729] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.935916 19758 consensus_queue.cc:260] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.936131 19766 tablet_bootstrap.cc:492] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:43.936184 19758 ts_tablet_manager.cc:1398] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.936203 19766 ts_tablet_manager.cc:1367] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:43.936151 19771 raft_consensus.cc:348] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.936237 19771 raft_consensus.cc:374] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.936266 19771 raft_consensus.cc:729] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.936408 19771 consensus_queue.cc:260] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.936710 19771 ts_tablet_manager.cc:1398] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:43.937501 19766 raft_consensus.cc:348] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.937593 19766 raft_consensus.cc:374] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.937620 19766 raft_consensus.cc:729] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.937718 19766 consensus_queue.cc:260] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:43.938069 19766 ts_tablet_manager.cc:1398] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:43.938223 19766 tablet_bootstrap.cc:492] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:43.939204 19766 tablet_bootstrap.cc:654] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:43.940339 19766 tablet_bootstrap.cc:492] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:43.937676 19767 ts_tablet_manager.cc:1398] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:43.940426 19766 ts_tablet_manager.cc:1367] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:43.940806 19766 raft_consensus.cc:348] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.940901 19766 raft_consensus.cc:374] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:43.940932 19766 raft_consensus.cc:729] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:43.941033 19766 consensus_queue.cc:260] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.941275 19766 ts_tablet_manager.cc:1398] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:43.958779 19552 raft_consensus.cc:482] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.958881 19552 raft_consensus.cc:504] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.959257 19552 leader_election.cc:288] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:43.959632 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff37bd5ee4274e1eb2f7e461aa3b57c1" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:43.959756 14958 raft_consensus.cc:2454] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:43.960008 15155 leader_election.cc:302] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:43.960062 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff37bd5ee4274e1eb2f7e461aa3b57c1" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:43.960204 12848 raft_consensus.cc:2454] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:43.960271 19552 raft_consensus.cc:2781] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:43.960340 19552 raft_consensus.cc:482] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.960291 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:43.960389 19552 raft_consensus.cc:3037] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.961336 19552 raft_consensus.cc:504] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.961666 19552 leader_election.cc:288] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:43.962152 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff37bd5ee4274e1eb2f7e461aa3b57c1" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:43.962275 14958 raft_consensus.cc:3037] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.963055 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff37bd5ee4274e1eb2f7e461aa3b57c1" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:43.963186 12848 raft_consensus.cc:3037] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.963320 14958 raft_consensus.cc:2454] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:43.963555 15155 leader_election.cc:302] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:43.963719 19552 raft_consensus.cc:2781] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.963798 19552 raft_consensus.cc:686] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:43.963968 19552 consensus_queue.cc:237] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.964267 12848 raft_consensus.cc:2454] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:43.966631 14958 raft_consensus.cc:1261] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:43.966547 12441 catalog_manager.cc:5484] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:43.967149 19684 consensus_queue.cc:1035] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:43.967528 12653 consensus_peers.cc:485] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:09:43.970180 19783 tablet.cc:1722] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.970510 19783 tablet.cc:1620] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:43.971822 19618 tablet.cc:1722] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:43.972110 19618 tablet.cc:1620] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:43.973510 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet ae21044b255848aab31cf7d3094cbe2f (table rebalancer_dynamic_table [id=0ce2423da7134e9d8e2e2a9c3288f968]). Expected version 1 got 0
I20240501 14:09:43.974076 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=0ce2423da7134e9d8e2e2a9c3288f968] alter complete (version 1)
I20240501 14:09:43.992313 19552 raft_consensus.cc:482] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:43.992444 19552 raft_consensus.cc:504] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.992856 19552 leader_election.cc:288] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:43.993371 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bf116648874fc3a3ff54c7addf5e2d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:43.993511 15351 raft_consensus.cc:2454] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:43.993769 15157 leader_election.cc:302] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:43.993932 19552 raft_consensus.cc:2781] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 14:09:43.994011 15156 leader_election.cc:334] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.994010 19552 raft_consensus.cc:482] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:43.994136 19552 raft_consensus.cc:3037] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:43.994964 19552 raft_consensus.cc:504] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.995280 19552 leader_election.cc:288] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673)
I20240501 14:09:43.995572 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bf116648874fc3a3ff54c7addf5e2d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:43.995724 15351 raft_consensus.cc:3037] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:43.996212 15156 leader_election.cc:334] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:43.996623 15351 raft_consensus.cc:2454] T c0bf116648874fc3a3ff54c7addf5e2d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:43.996902 15157 leader_election.cc:302] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:43.997144 19552 raft_consensus.cc:2781] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:43.997236 19552 raft_consensus.cc:686] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:43.997372 19552 consensus_queue.cc:237] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:43.999656 12441 catalog_manager.cc:5484] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:44.075641 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:44.085037 19783 tablet.cc:1722] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.085824 19618 tablet.cc:1722] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.086120 19618 tablet.cc:1620] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:44.086170 19783 tablet.cc:1620] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:44.091182 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=0ce2423da7134e9d8e2e2a9c3288f968] alter complete (version 2)
W20240501 14:09:44.159473 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): CreateTablet RPC failed for tablet ae21044b255848aab31cf7d3094cbe2f: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:44.194077 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:44.199208 19783 tablet.cc:1722] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.199641 19618 tablet.cc:1722] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.204272 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=0ce2423da7134e9d8e2e2a9c3288f968] alter complete (version 3)
W20240501 14:09:44.238284 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.306635 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:44.306826 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:44.308785 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet ae21044b255848aab31cf7d3094cbe2f
I20240501 14:09:44.309728 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet ae21044b255848aab31cf7d3094cbe2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:44.309892 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet ae21044b255848aab31cf7d3094cbe2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:44.310240 19802 tablet_replica.cc:330] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:44.310266 19801 tablet_replica.cc:330] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:44.310400 19801 raft_consensus.cc:2229] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.310458 19802 raft_consensus.cc:2229] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:44.310631 19801 raft_consensus.cc:2258] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.310772 19802 raft_consensus.cc:2258] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.311337 19802 ts_tablet_manager.cc:1875] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.311441 19801 ts_tablet_manager.cc:1875] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.313306 19801 ts_tablet_manager.cc:1888] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.313308 19802 ts_tablet_manager.cc:1888] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.313369 19801 log.cc:1196] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/ae21044b255848aab31cf7d3094cbe2f
I20240501 14:09:44.313374 19802 log.cc:1196] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/ae21044b255848aab31cf7d3094cbe2f
I20240501 14:09:44.313627 19801 ts_tablet_manager.cc:1909] T ae21044b255848aab31cf7d3094cbe2f P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:44.313668 19802 ts_tablet_manager.cc:1909] T ae21044b255848aab31cf7d3094cbe2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:44.314321 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet ae21044b255848aab31cf7d3094cbe2f (table rebalancer_dynamic_table_new_name [id=0ce2423da7134e9d8e2e2a9c3288f968]) successfully deleted
I20240501 14:09:44.314360 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet ae21044b255848aab31cf7d3094cbe2f (table rebalancer_dynamic_table_new_name [id=0ce2423da7134e9d8e2e2a9c3288f968]) successfully deleted
I20240501 14:09:44.371042 19803 raft_consensus.cc:482] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.371227 19803 raft_consensus.cc:504] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:44.371762 19803 leader_election.cc:288] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:44.372778 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6e2acdbdc747a8a863d6e46d0d1b41" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:44.373023 15221 raft_consensus.cc:2454] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:44.373383 14891 leader_election.cc:302] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
W20240501 14:09:44.373786 14893 leader_election.cc:334] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:44.373829 19803 raft_consensus.cc:2781] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.373898 19803 raft_consensus.cc:482] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.373946 19803 raft_consensus.cc:3037] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.375252 19803 raft_consensus.cc:504] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:44.375572 19803 leader_election.cc:288] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:44.375994 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6e2acdbdc747a8a863d6e46d0d1b41" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:44.376119 15221 raft_consensus.cc:3037] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:44.376652 14893 leader_election.cc:334] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:44.377146 15221 raft_consensus.cc:2454] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:44.377455 14891 leader_election.cc:302] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters: ab90e77f0027449bb3155980ca95593c
I20240501 14:09:44.377687 19803 raft_consensus.cc:2781] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.377775 19803 raft_consensus.cc:686] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:44.377912 19803 consensus_queue.cc:237] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:44.379864 12443 catalog_manager.cc:5484] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:44.410774 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:44.416280 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 26b7b1e805c34877b8b66a5c8409c6f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc]), partition=
I20240501 14:09:44.416512 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 26b7b1e805c34877b8b66a5c8409c6f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc]), partition=
I20240501 14:09:44.416574 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7b1e805c34877b8b66a5c8409c6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.416786 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7b1e805c34877b8b66a5c8409c6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.418609 19758 tablet_bootstrap.cc:492] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
W20240501 14:09:44.418587 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.419090 19817 tablet_bootstrap.cc:492] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:44.419135 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 26b7b1e805c34877b8b66a5c8409c6f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc]), partition=
I20240501 14:09:44.419371 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b7b1e805c34877b8b66a5c8409c6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.419584 19758 tablet_bootstrap.cc:654] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.420282 19817 tablet_bootstrap.cc:654] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.420599 19758 tablet_bootstrap.cc:492] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:44.420684 19758 ts_tablet_manager.cc:1367] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.421178 19758 raft_consensus.cc:348] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.421274 19758 raft_consensus.cc:374] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.421316 19758 raft_consensus.cc:729] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.421432 19758 consensus_queue.cc:260] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.421635 19767 tablet_bootstrap.cc:492] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:44.421711 19758 ts_tablet_manager.cc:1398] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.421830 19817 tablet_bootstrap.cc:492] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:44.422531 19767 tablet_bootstrap.cc:654] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.422722 19817 ts_tablet_manager.cc:1367] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:44.423801 19767 tablet_bootstrap.cc:492] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:44.423889 19767 ts_tablet_manager.cc:1367] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:44.424278 19767 raft_consensus.cc:348] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.424371 19767 raft_consensus.cc:374] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.424407 19767 raft_consensus.cc:729] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.424535 19767 consensus_queue.cc:260] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.424618 19817 raft_consensus.cc:348] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.424805 19767 ts_tablet_manager.cc:1398] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.424723 19817 raft_consensus.cc:374] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.424894 19817 raft_consensus.cc:729] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.425024 19817 consensus_queue.cc:260] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.425292 19817 ts_tablet_manager.cc:1398] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 14:09:44.432191 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:44.443152 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.456259 18656 raft_consensus.cc:482] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.456378 18656 raft_consensus.cc:504] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.456758 18656 leader_election.cc:288] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:44.457103 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7b1e805c34877b8b66a5c8409c6f5" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:44.457244 14958 raft_consensus.cc:2454] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:44.457275 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7b1e805c34877b8b66a5c8409c6f5" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:44.457405 15353 raft_consensus.cc:2454] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:44.457543 12652 leader_election.cc:302] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:44.457811 18656 raft_consensus.cc:2781] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.457864 18656 raft_consensus.cc:482] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.457906 18656 raft_consensus.cc:3037] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.458850 18656 raft_consensus.cc:504] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.459188 18656 leader_election.cc:288] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:44.459547 15353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7b1e805c34877b8b66a5c8409c6f5" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:44.459553 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26b7b1e805c34877b8b66a5c8409c6f5" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:44.459664 15353 raft_consensus.cc:3037] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.459667 14958 raft_consensus.cc:3037] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.460592 15353 raft_consensus.cc:2454] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:44.460651 14958 raft_consensus.cc:2454] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:44.460915 12652 leader_election.cc:302] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:44.461066 18656 raft_consensus.cc:2781] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.461179 18656 raft_consensus.cc:686] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:44.461328 18656 consensus_queue.cc:237] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.463399 12443 catalog_manager.cc:5484] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:44.479128 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.501677 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:44.515726 19552 consensus_queue.cc:1035] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:44.516273 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.518920 19823 consensus_queue.cc:1035] T ff37bd5ee4274e1eb2f7e461aa3b57c1 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.570777 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:44.574506 14957 raft_consensus.cc:1261] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.574564 15353 raft_consensus.cc:1261] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:44.575029 18656 consensus_queue.cc:1035] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.575162 19685 consensus_queue.cc:1035] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.577214 19783 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.577487 19783 tablet.cc:1620] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:44.578094 19618 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.578336 19618 tablet.cc:1620] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:44.581506 12439 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 26b7b1e805c34877b8b66a5c8409c6f5 (table rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc]). Expected version 1 got 0
I20240501 14:09:44.581830 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 26b7b1e805c34877b8b66a5c8409c6f5 (table rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc]). Expected version 1 got 0
I20240501 14:09:44.581981 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc] alter complete (version 1)
I20240501 14:09:44.585377 19830 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:44.585671 19830 tablet.cc:1620] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:44.586715 12439 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 26b7b1e805c34877b8b66a5c8409c6f5 (table rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc]). Expected version 1 got 0
I20240501 14:09:44.607709 19823 consensus_queue.cc:1035] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:44.627069 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.629967 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.630740 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.645220 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.664151 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.678050 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.683388 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.688884 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:44.694190 19783 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.694555 19783 tablet.cc:1620] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:44.694723 19830 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.695027 19830 tablet.cc:1620] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:44.695339 19618 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:44.695644 19618 tablet.cc:1620] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:44.700450 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=1b5a585c087b462fb32665ee417407fc] alter complete (version 2)
W20240501 14:09:44.712697 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.717234 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.804066 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:44.809962 19783 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.810428 19618 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.813628 19830 tablet.cc:1722] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:44.815496 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=1b5a585c087b462fb32665ee417407fc] alter complete (version 3)
W20240501 14:09:44.840790 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:09:44.861043 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:44.884006 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.887215 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:44.892388 19803 consensus_queue.cc:1035] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:44.893550 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 02c06482797841aab99e23dfab1e21de (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.893873 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c06482797841aab99e23dfab1e21de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.894286 12566 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b101a4f42413695212c658bc2a260 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.894568 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b101a4f42413695212c658bc2a260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.894856 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 02c06482797841aab99e23dfab1e21de (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.894961 15202 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b101a4f42413695212c658bc2a260 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.895009 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 787f22e3c76140faa2923faca97ba57e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.895090 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c06482797841aab99e23dfab1e21de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.895257 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787f22e3c76140faa2923faca97ba57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.895383 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 02c06482797841aab99e23dfab1e21de (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.895684 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c06482797841aab99e23dfab1e21de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.897433 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 787f22e3c76140faa2923faca97ba57e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.897603 19817 tablet_bootstrap.cc:492] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:44.897620 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b101a4f42413695212c658bc2a260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.897768 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787f22e3c76140faa2923faca97ba57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.898500 19817 tablet_bootstrap.cc:654] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.899914 19853 tablet_bootstrap.cc:492] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:44.900422 15333 tablet_service.cc:1467] Processing CreateTablet for tablet 787f22e3c76140faa2923faca97ba57e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.900568 19817 tablet_bootstrap.cc:492] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:44.900642 19817 ts_tablet_manager.cc:1367] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:44.900674 15333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787f22e3c76140faa2923faca97ba57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.900832 15332 tablet_service.cc:1467] Processing CreateTablet for tablet bb4b101a4f42413695212c658bc2a260 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=e267c6cc548c422ea1755d3cc82f2fa3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:44.901039 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4b101a4f42413695212c658bc2a260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:44.901043 19817 raft_consensus.cc:348] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.901124 19817 raft_consensus.cc:374] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.901144 19817 raft_consensus.cc:729] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.901239 19817 consensus_queue.cc:260] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.901520 19817 ts_tablet_manager.cc:1398] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:44.901633 19817 tablet_bootstrap.cc:492] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:44.902271 19817 tablet_bootstrap.cc:654] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.902897 19767 tablet_bootstrap.cc:492] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:44.903808 19767 tablet_bootstrap.cc:654] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.906672 19853 tablet_bootstrap.cc:654] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.908509 19854 tablet_bootstrap.cc:492] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:44.908775 19767 tablet_bootstrap.cc:492] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:44.908861 19767 ts_tablet_manager.cc:1367] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:44.909358 19767 raft_consensus.cc:348] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.909473 19767 raft_consensus.cc:374] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.909508 19767 raft_consensus.cc:729] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.909631 19767 consensus_queue.cc:260] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.909809 19817 tablet_bootstrap.cc:492] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:44.909894 19817 ts_tablet_manager.cc:1367] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:09:44.909897 19767 ts_tablet_manager.cc:1398] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.910039 19853 tablet_bootstrap.cc:492] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:44.910065 19767 tablet_bootstrap.cc:492] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:44.910117 19853 ts_tablet_manager.cc:1367] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:44.910382 19817 raft_consensus.cc:348] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.910486 19817 raft_consensus.cc:374] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.910516 19817 raft_consensus.cc:729] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.910570 19853 raft_consensus.cc:348] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.910625 19817 consensus_queue.cc:260] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.910696 19853 raft_consensus.cc:374] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.910725 19853 raft_consensus.cc:729] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.910897 19817 ts_tablet_manager.cc:1398] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.910890 19853 consensus_queue.cc:260] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.911144 19853 ts_tablet_manager.cc:1398] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:44.912585 19767 tablet_bootstrap.cc:654] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.909428 19854 tablet_bootstrap.cc:654] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.913758 19767 tablet_bootstrap.cc:492] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:44.913841 19767 ts_tablet_manager.cc:1367] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:44.914129 19854 tablet_bootstrap.cc:492] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:44.914274 19767 raft_consensus.cc:348] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.914346 19767 raft_consensus.cc:374] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.914346 19854 ts_tablet_manager.cc:1367] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:44.914366 19767 raft_consensus.cc:729] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.914469 19767 consensus_queue.cc:260] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.914706 19767 ts_tablet_manager.cc:1398] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:44.914889 19767 tablet_bootstrap.cc:492] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:44.914953 19854 raft_consensus.cc:348] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.915333 19854 raft_consensus.cc:374] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.915422 19854 raft_consensus.cc:729] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.915618 19854 consensus_queue.cc:260] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.915721 19767 tablet_bootstrap.cc:654] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.915917 19854 ts_tablet_manager.cc:1398] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.916203 19854 tablet_bootstrap.cc:492] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:44.916854 19767 tablet_bootstrap.cc:492] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:44.916934 19767 ts_tablet_manager.cc:1367] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:44.917343 19767 raft_consensus.cc:348] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:44.917452 19767 raft_consensus.cc:374] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.917487 19767 raft_consensus.cc:729] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.917587 19767 consensus_queue.cc:260] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
W20240501 14:09:44.917750 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.917851 19767 ts_tablet_manager.cc:1398] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:44.918241 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:44.918365 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:44.918965 19854 tablet_bootstrap.cc:654] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.922029 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 26b7b1e805c34877b8b66a5c8409c6f5
I20240501 14:09:44.922461 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b7b1e805c34877b8b66a5c8409c6f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:44.922542 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b7b1e805c34877b8b66a5c8409c6f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:44.922852 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 26b7b1e805c34877b8b66a5c8409c6f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:44 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:44.923048 19866 tablet_replica.cc:330] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:44.923146 19867 tablet_replica.cc:330] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:44.923213 19866 raft_consensus.cc:2229] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.923291 19867 raft_consensus.cc:2229] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:44.923338 19854 tablet_bootstrap.cc:492] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:44.923332 19866 raft_consensus.cc:2258] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.923414 19854 ts_tablet_manager.cc:1367] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 14:09:44.923514 19867 raft_consensus.cc:2258] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.923952 19866 ts_tablet_manager.cc:1875] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.924023 19867 ts_tablet_manager.cc:1875] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.924022 19854 raft_consensus.cc:348] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.924154 19854 raft_consensus.cc:374] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.924199 19854 raft_consensus.cc:729] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.924314 19854 consensus_queue.cc:260] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.925344 19854 ts_tablet_manager.cc:1398] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:44.925558 19854 tablet_bootstrap.cc:492] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
W20240501 14:09:44.926527 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:44.926671 19854 tablet_bootstrap.cc:654] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:44.927366 19865 tablet_replica.cc:330] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:44.927531 19866 ts_tablet_manager.cc:1888] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.927562 19865 raft_consensus.cc:2229] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:44.927668 19866 log.cc:1196] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/26b7b1e805c34877b8b66a5c8409c6f5
I20240501 14:09:44.927729 19865 raft_consensus.cc:2258] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:44.927839 19867 ts_tablet_manager.cc:1888] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.927912 19867 log.cc:1196] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/26b7b1e805c34877b8b66a5c8409c6f5
I20240501 14:09:44.928095 19866 ts_tablet_manager.cc:1909] T 26b7b1e805c34877b8b66a5c8409c6f5 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:44.928265 19867 ts_tablet_manager.cc:1909] T 26b7b1e805c34877b8b66a5c8409c6f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:44.928936 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 26b7b1e805c34877b8b66a5c8409c6f5 (table rebalancer_dynamic_table_new_name [id=1b5a585c087b462fb32665ee417407fc]) successfully deleted
I20240501 14:09:44.929260 19854 tablet_bootstrap.cc:492] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:44.929283 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 26b7b1e805c34877b8b66a5c8409c6f5 (table rebalancer_dynamic_table_new_name [id=1b5a585c087b462fb32665ee417407fc]) successfully deleted
I20240501 14:09:44.929347 19854 ts_tablet_manager.cc:1367] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:44.929279 19865 ts_tablet_manager.cc:1875] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:44.929812 19854 raft_consensus.cc:348] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.929930 19854 raft_consensus.cc:374] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:44.929957 19854 raft_consensus.cc:729] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:44.930030 18520 raft_consensus.cc:482] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.930109 19854 consensus_queue.cc:260] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.930120 18520 raft_consensus.cc:504] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.930359 19854 ts_tablet_manager.cc:1398] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:44.930588 18520 leader_election.cc:288] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:44.930985 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b101a4f42413695212c658bc2a260" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:44.931080 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b101a4f42413695212c658bc2a260" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:44.931129 12586 raft_consensus.cc:2454] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:44.931185 15222 raft_consensus.cc:2454] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:44.931474 15286 leader_election.cc:302] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:44.931730 19865 ts_tablet_manager.cc:1888] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:44.931807 18520 raft_consensus.cc:2781] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.931830 19865 log.cc:1196] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/26b7b1e805c34877b8b66a5c8409c6f5
I20240501 14:09:44.931905 18520 raft_consensus.cc:482] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.931949 18520 raft_consensus.cc:3037] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.932113 19865 ts_tablet_manager.cc:1909] T 26b7b1e805c34877b8b66a5c8409c6f5 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:44.932811 18520 raft_consensus.cc:504] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.932864 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 26b7b1e805c34877b8b66a5c8409c6f5 (table rebalancer_dynamic_table_new_name [id=1b5a585c087b462fb32665ee417407fc]) successfully deleted
I20240501 14:09:44.933143 18520 leader_election.cc:288] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:44.933434 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b101a4f42413695212c658bc2a260" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:44.933549 15222 raft_consensus.cc:3037] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.933532 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4b101a4f42413695212c658bc2a260" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:44.933640 12586 raft_consensus.cc:3037] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.934499 15222 raft_consensus.cc:2454] T bb4b101a4f42413695212c658bc2a260 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:44.934597 12586 raft_consensus.cc:2454] T bb4b101a4f42413695212c658bc2a260 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:44.934942 15288 leader_election.cc:302] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:44.935175 18520 raft_consensus.cc:2781] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.935262 18520 raft_consensus.cc:686] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:44.935417 18520 consensus_queue.cc:237] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:44.937579 12441 catalog_manager.cc:5484] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:44.959558 19584 raft_consensus.cc:482] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:44.959686 19584 raft_consensus.cc:504] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.960099 19584 leader_election.cc:288] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:44.960448 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787f22e3c76140faa2923faca97ba57e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:44.960456 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787f22e3c76140faa2923faca97ba57e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:44.960628 15351 raft_consensus.cc:2454] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:44.960623 15222 raft_consensus.cc:2454] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:44.960914 12651 leader_election.cc:302] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:44.961089 19584 raft_consensus.cc:2781] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:44.961148 19584 raft_consensus.cc:482] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:44.961182 19584 raft_consensus.cc:3037] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.962046 19584 raft_consensus.cc:504] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.962355 19584 leader_election.cc:288] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:44.962661 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787f22e3c76140faa2923faca97ba57e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:44.962810 15351 raft_consensus.cc:3037] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.962797 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787f22e3c76140faa2923faca97ba57e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:44.962919 15222 raft_consensus.cc:3037] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:44.963569 15351 raft_consensus.cc:2454] T 787f22e3c76140faa2923faca97ba57e P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:44.963682 15222 raft_consensus.cc:2454] T 787f22e3c76140faa2923faca97ba57e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:44.963877 12654 leader_election.cc:302] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:44.964028 19584 raft_consensus.cc:2781] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:44.964128 19584 raft_consensus.cc:686] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:44.964289 19584 consensus_queue.cc:237] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:44.966267 12441 catalog_manager.cc:5484] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:45.004374 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:45.023581 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:45.028556 12827 tablet_service.cc:1467] Processing CreateTablet for tablet b7ac69364b934a92b08e80add4e565b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17]), partition=
I20240501 14:09:45.028621 15331 tablet_service.cc:1467] Processing CreateTablet for tablet b7ac69364b934a92b08e80add4e565b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17]), partition=
I20240501 14:09:45.028620 14937 tablet_service.cc:1467] Processing CreateTablet for tablet b7ac69364b934a92b08e80add4e565b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17]), partition=
I20240501 14:09:45.028848 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ac69364b934a92b08e80add4e565b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.028868 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ac69364b934a92b08e80add4e565b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.028868 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7ac69364b934a92b08e80add4e565b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.030710 19872 tablet_bootstrap.cc:492] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:45.031005 19767 tablet_bootstrap.cc:492] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:45.031716 19872 tablet_bootstrap.cc:654] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.031968 19767 tablet_bootstrap.cc:654] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.031989 19873 tablet_bootstrap.cc:492] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:45.033046 19767 tablet_bootstrap.cc:492] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:45.033130 19767 ts_tablet_manager.cc:1367] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:45.033368 19873 tablet_bootstrap.cc:654] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.033607 19767 raft_consensus.cc:348] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.033715 19767 raft_consensus.cc:374] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.033746 19767 raft_consensus.cc:729] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.033895 19767 consensus_queue.cc:260] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.034013 19872 tablet_bootstrap.cc:492] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:45.034108 19872 ts_tablet_manager.cc:1367] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:45.034230 19767 ts_tablet_manager.cc:1398] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.034396 19873 tablet_bootstrap.cc:492] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:45.034489 19873 ts_tablet_manager.cc:1367] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:45.034520 19872 raft_consensus.cc:348] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.034705 19872 raft_consensus.cc:374] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.034773 19872 raft_consensus.cc:729] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.034955 19872 consensus_queue.cc:260] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.035286 19872 ts_tablet_manager.cc:1398] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.035291 19873 raft_consensus.cc:348] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.035388 19873 raft_consensus.cc:374] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.035421 19873 raft_consensus.cc:729] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.035518 19873 consensus_queue.cc:260] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.035741 19873 ts_tablet_manager.cc:1398] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.053287 19584 raft_consensus.cc:482] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.053408 19584 raft_consensus.cc:504] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.053798 19584 leader_election.cc:288] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:45.054455 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c06482797841aab99e23dfab1e21de" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:45.054595 15222 raft_consensus.cc:2454] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:45.054898 12651 leader_election.cc:302] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:45.055284 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c06482797841aab99e23dfab1e21de" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:45.055433 15351 raft_consensus.cc:2454] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:45.055632 19584 raft_consensus.cc:2781] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.055687 19584 raft_consensus.cc:482] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.055724 19584 raft_consensus.cc:3037] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.056880 19584 raft_consensus.cc:504] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.057215 19584 leader_election.cc:288] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:45.057581 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c06482797841aab99e23dfab1e21de" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:45.057690 15222 raft_consensus.cc:3037] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.058189 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c06482797841aab99e23dfab1e21de" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:45.058292 15352 raft_consensus.cc:3037] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.058729 15222 raft_consensus.cc:2454] T 02c06482797841aab99e23dfab1e21de P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:45.059028 12651 leader_election.cc:302] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:45.059600 15352 raft_consensus.cc:2454] T 02c06482797841aab99e23dfab1e21de P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:45.060616 19584 raft_consensus.cc:2781] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.060710 19584 raft_consensus.cc:686] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:45.060858 19584 consensus_queue.cc:237] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.063067 12441 catalog_manager.cc:5484] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:45.125633 19549 raft_consensus.cc:482] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.125761 19549 raft_consensus.cc:504] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.126197 19549 leader_election.cc:288] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:45.126523 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ac69364b934a92b08e80add4e565b3" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:45.126541 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ac69364b934a92b08e80add4e565b3" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:45.126650 15352 raft_consensus.cc:2454] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:45.126684 14957 raft_consensus.cc:2454] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:45.127023 12783 leader_election.cc:302] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:45.127175 19549 raft_consensus.cc:2781] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.127228 19549 raft_consensus.cc:482] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.127278 19549 raft_consensus.cc:3037] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.128286 19549 raft_consensus.cc:504] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.128584 19549 leader_election.cc:288] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:45.128870 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ac69364b934a92b08e80add4e565b3" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:45.128897 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7ac69364b934a92b08e80add4e565b3" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:45.128978 14957 raft_consensus.cc:3037] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.129019 15352 raft_consensus.cc:3037] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.129724 14957 raft_consensus.cc:2454] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:45.129727 15352 raft_consensus.cc:2454] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:45.130033 12783 leader_election.cc:302] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:45.130200 19549 raft_consensus.cc:2781] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.130388 19549 raft_consensus.cc:686] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:45.130779 19549 consensus_queue.cc:237] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:45.132776 12441 catalog_manager.cc:5484] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.257031 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:45.261215 15352 raft_consensus.cc:1261] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.261374 14957 raft_consensus.cc:1261] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:45.261685 19549 consensus_queue.cc:1035] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.261878 19888 consensus_queue.cc:1035] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.264879 19893 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:45.265197 19893 tablet.cc:1620] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:45.266912 12441 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet b7ac69364b934a92b08e80add4e565b3 (table rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17]). Expected version 1 got 0
I20240501 14:09:45.267146 19830 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:45.267433 19830 tablet.cc:1620] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:45.268936 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet b7ac69364b934a92b08e80add4e565b3 (table rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17]). Expected version 1 got 0
I20240501 14:09:45.271472 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17] alter complete (version 1)
I20240501 14:09:45.271697 19618 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:45.272002 19618 tablet.cc:1620] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:45.272892 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet b7ac69364b934a92b08e80add4e565b3 (table rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17]). Expected version 1 got 0
I20240501 14:09:45.334916 19860 consensus_queue.cc:1035] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.338984 19860 consensus_queue.cc:1035] T bb4b101a4f42413695212c658bc2a260 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.361704 18656 consensus_queue.cc:1035] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.364717 19685 consensus_queue.cc:1035] T 787f22e3c76140faa2923faca97ba57e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.376621 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:45.385309 19893 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:45.385622 19893 tablet.cc:1620] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:45.385732 19618 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:45.386020 19618 tablet.cc:1620] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:45.391558 19830 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:45.391891 19830 tablet.cc:1620] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:45.391883 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=99e6952ac13249d2b31d0e42b9417e17] alter complete (version 2)
W20240501 14:09:45.424685 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae21044b255848aab31cf7d3094cbe2f: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:45.493290 19685 consensus_queue.cc:1035] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.494774 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:45.497829 19584 consensus_queue.cc:1035] T 02c06482797841aab99e23dfab1e21de P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:45.502285 19893 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:45.502758 19830 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:45.502851 19618 tablet.cc:1722] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:45.509630 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=99e6952ac13249d2b31d0e42b9417e17] alter complete (version 3)
W20240501 14:09:45.565768 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:45.567579 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:45.574214 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 64d71f1f43ce4a918090e94a3679c062 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.574214 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 64d71f1f43ce4a918090e94a3679c062 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.574316 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 294cc079a8344f6abe0002063a436e77 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.574525 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d71f1f43ce4a918090e94a3679c062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.574533 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d71f1f43ce4a918090e94a3679c062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.574644 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294cc079a8344f6abe0002063a436e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.574934 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 64d71f1f43ce4a918090e94a3679c062 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.575207 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d71f1f43ce4a918090e94a3679c062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.577564 19928 tablet_bootstrap.cc:492] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:45.577579 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 294cc079a8344f6abe0002063a436e77 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.577638 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 294cc079a8344f6abe0002063a436e77 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.577857 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294cc079a8344f6abe0002063a436e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.577889 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294cc079a8344f6abe0002063a436e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.579051 19928 tablet_bootstrap.cc:654] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.579720 19927 tablet_bootstrap.cc:492] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:45.580695 19927 tablet_bootstrap.cc:654] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.580757 19930 tablet_bootstrap.cc:492] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:45.581888 19925 tablet_bootstrap.cc:492] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:45.581879 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 7a487ec3ad684730b082a278dbf23045 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.582109 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a487ec3ad684730b082a278dbf23045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.582931 19925 tablet_bootstrap.cc:654] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.583340 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 7a487ec3ad684730b082a278dbf23045 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.583571 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a487ec3ad684730b082a278dbf23045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.583647 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 7a487ec3ad684730b082a278dbf23045 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=020f483f50e444829bf231f52830604a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:45.583886 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a487ec3ad684730b082a278dbf23045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.585070 19931 tablet_bootstrap.cc:492] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:45.586019 19931 tablet_bootstrap.cc:654] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.586186 19930 tablet_bootstrap.cc:654] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.587141 19925 tablet_bootstrap.cc:492] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:45.587280 19927 tablet_bootstrap.cc:492] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:45.587272 19925 ts_tablet_manager.cc:1367] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:45.587438 19927 ts_tablet_manager.cc:1367] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:45.587657 19926 tablet_bootstrap.cc:492] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:45.587734 19925 raft_consensus.cc:348] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.587932 19925 raft_consensus.cc:374] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.588027 19925 raft_consensus.cc:729] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.588279 19925 consensus_queue.cc:260] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.588642 19925 ts_tablet_manager.cc:1398] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.589213 19927 raft_consensus.cc:348] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.589288 19925 tablet_bootstrap.cc:492] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:45.589323 19927 raft_consensus.cc:374] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.589355 19927 raft_consensus.cc:729] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.589481 19927 consensus_queue.cc:260] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.590212 19925 tablet_bootstrap.cc:654] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.591840 19926 tablet_bootstrap.cc:654] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.592927 19930 tablet_bootstrap.cc:492] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:45.593015 19930 ts_tablet_manager.cc:1367] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 14:09:45.593243 19927 ts_tablet_manager.cc:1398] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:45.593497 19930 raft_consensus.cc:348] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.593633 19930 raft_consensus.cc:374] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.593679 19930 raft_consensus.cc:729] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.593811 19930 consensus_queue.cc:260] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.594136 19930 ts_tablet_manager.cc:1398] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.595060 19925 tablet_bootstrap.cc:492] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:45.595137 19925 ts_tablet_manager.cc:1367] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:45.595772 19925 raft_consensus.cc:348] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.595908 19925 raft_consensus.cc:374] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.595974 19925 raft_consensus.cc:729] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.596205 19933 tablet_bootstrap.cc:492] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:45.596809 19925 consensus_queue.cc:260] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.597252 19925 ts_tablet_manager.cc:1398] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:45.597435 19933 tablet_bootstrap.cc:654] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.597879 19931 tablet_bootstrap.cc:492] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:45.598023 19931 ts_tablet_manager.cc:1367] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:45.598632 19931 raft_consensus.cc:348] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.598842 19931 raft_consensus.cc:374] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.598914 19931 raft_consensus.cc:729] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.599077 19931 consensus_queue.cc:260] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.599558 19931 ts_tablet_manager.cc:1398] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:45.600841 19933 tablet_bootstrap.cc:492] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:45.600930 19933 ts_tablet_manager.cc:1367] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:45.601416 19933 raft_consensus.cc:348] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.601521 19933 raft_consensus.cc:374] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.601565 19933 raft_consensus.cc:729] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.606932 19933 consensus_queue.cc:260] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.607236 19933 ts_tablet_manager.cc:1398] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:09:45.607959 19928 tablet_bootstrap.cc:492] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:45.608062 19926 tablet_bootstrap.cc:492] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:45.608067 19928 ts_tablet_manager.cc:1367] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240501 14:09:45.608150 19926 ts_tablet_manager.cc:1367] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240501 14:09:45.608500 19928 raft_consensus.cc:348] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.608654 19928 raft_consensus.cc:374] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.608635 19926 raft_consensus.cc:348] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.608698 19928 raft_consensus.cc:729] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.608726 19926 raft_consensus.cc:374] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.608752 19926 raft_consensus.cc:729] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.608821 19928 consensus_queue.cc:260] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.608847 19926 consensus_queue.cc:260] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.609095 19926 ts_tablet_manager.cc:1398] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.609189 19928 ts_tablet_manager.cc:1398] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.611348 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:45.611454 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:45.613862 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet b7ac69364b934a92b08e80add4e565b3
I20240501 14:09:45.614440 19928 tablet_bootstrap.cc:492] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:45.614606 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ac69364b934a92b08e80add4e565b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:45 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:45.615181 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ac69364b934a92b08e80add4e565b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:45 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:45.615535 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet b7ac69364b934a92b08e80add4e565b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:45 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:45.616248 19928 tablet_bootstrap.cc:654] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.619814 19928 tablet_bootstrap.cc:492] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:45.619911 19928 ts_tablet_manager.cc:1367] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:45.620384 19928 raft_consensus.cc:348] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.620535 19928 raft_consensus.cc:374] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.620594 19928 raft_consensus.cc:729] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.620720 19928 consensus_queue.cc:260] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.621011 19928 ts_tablet_manager.cc:1398] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:45.621237 19942 tablet_replica.cc:330] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:45.621399 19942 raft_consensus.cc:2229] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:45.621631 19942 raft_consensus.cc:2258] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.621879 19943 tablet_replica.cc:330] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:45.622087 19943 raft_consensus.cc:2229] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.622179 19942 ts_tablet_manager.cc:1875] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:45.622239 19943 raft_consensus.cc:2258] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.622599 19941 tablet_replica.cc:330] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:45.622787 19941 raft_consensus.cc:2229] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:45.622915 19941 raft_consensus.cc:2258] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:45.623438 19941 ts_tablet_manager.cc:1875] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:45.623374 19943 ts_tablet_manager.cc:1875] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:45.625624 19942 ts_tablet_manager.cc:1888] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:45.625686 19943 ts_tablet_manager.cc:1888] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:45.625756 19942 log.cc:1196] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/b7ac69364b934a92b08e80add4e565b3
I20240501 14:09:45.625859 19943 log.cc:1196] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/b7ac69364b934a92b08e80add4e565b3
I20240501 14:09:45.626156 19942 ts_tablet_manager.cc:1909] T b7ac69364b934a92b08e80add4e565b3 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:45.626181 19943 ts_tablet_manager.cc:1909] T b7ac69364b934a92b08e80add4e565b3 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:45.626811 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet b7ac69364b934a92b08e80add4e565b3 (table rebalancer_dynamic_table_new_name [id=99e6952ac13249d2b31d0e42b9417e17]) successfully deleted
I20240501 14:09:45.626976 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet b7ac69364b934a92b08e80add4e565b3 (table rebalancer_dynamic_table_new_name [id=99e6952ac13249d2b31d0e42b9417e17]) successfully deleted
I20240501 14:09:45.632598 19941 ts_tablet_manager.cc:1888] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:45.632684 19941 log.cc:1196] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/b7ac69364b934a92b08e80add4e565b3
I20240501 14:09:45.632943 19941 ts_tablet_manager.cc:1909] T b7ac69364b934a92b08e80add4e565b3 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:45.633740 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet b7ac69364b934a92b08e80add4e565b3 (table rebalancer_dynamic_table_new_name [id=99e6952ac13249d2b31d0e42b9417e17]) successfully deleted
I20240501 14:09:45.671936 19940 raft_consensus.cc:482] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.672097 19940 raft_consensus.cc:504] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.672561 19940 leader_election.cc:288] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:45.672874 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d71f1f43ce4a918090e94a3679c062" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:45.672935 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d71f1f43ce4a918090e94a3679c062" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:45.673045 14957 raft_consensus.cc:2454] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:45.673022 15221 raft_consensus.cc:2454] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:45.673317 12520 leader_election.cc:302] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:45.673482 19940 raft_consensus.cc:2781] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.673552 19940 raft_consensus.cc:482] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.673591 19940 raft_consensus.cc:3037] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.674654 19940 raft_consensus.cc:504] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.674993 19940 leader_election.cc:288] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:45.675297 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d71f1f43ce4a918090e94a3679c062" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:45.675297 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d71f1f43ce4a918090e94a3679c062" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:45.675423 15221 raft_consensus.cc:3037] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.675416 14957 raft_consensus.cc:3037] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.676367 15221 raft_consensus.cc:2454] T 64d71f1f43ce4a918090e94a3679c062 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:45.676367 14957 raft_consensus.cc:2454] T 64d71f1f43ce4a918090e94a3679c062 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:45.676667 12521 leader_election.cc:302] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:45.676952 19940 raft_consensus.cc:2781] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.677042 19940 raft_consensus.cc:686] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:45.677182 19940 consensus_queue.cc:237] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:45.679311 12441 catalog_manager.cc:5484] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.715595 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:45.720898 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]), partition=
I20240501 14:09:45.721128 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]), partition=
I20240501 14:09:45.721199 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.721211 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]), partition=
I20240501 14:09:45.721349 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.721405 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:45.723208 19927 tablet_bootstrap.cc:492] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:45.723517 19925 tablet_bootstrap.cc:492] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:45.724421 19927 tablet_bootstrap.cc:654] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.724424 19925 tablet_bootstrap.cc:654] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.725723 19927 tablet_bootstrap.cc:492] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:45.725804 19927 ts_tablet_manager.cc:1367] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:45.726094 19925 tablet_bootstrap.cc:492] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:45.726167 19925 ts_tablet_manager.cc:1367] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:09:45.726189 19927 raft_consensus.cc:348] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:45.726284 19927 raft_consensus.cc:374] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.726326 19927 raft_consensus.cc:729] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.726374 19933 tablet_bootstrap.cc:492] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:45.726437 19927 consensus_queue.cc:260] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:45.726588 19925 raft_consensus.cc:348] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:45.726684 19925 raft_consensus.cc:374] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.726691 19927 ts_tablet_manager.cc:1398] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:45.726716 19925 raft_consensus.cc:729] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.726871 19925 consensus_queue.cc:260] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:45.727121 19925 ts_tablet_manager.cc:1398] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.727294 19933 tablet_bootstrap.cc:654] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:45.729532 19933 tablet_bootstrap.cc:492] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:45.729624 19933 ts_tablet_manager.cc:1367] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:45.730060 19933 raft_consensus.cc:348] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:45.730168 19933 raft_consensus.cc:374] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:45.730195 19933 raft_consensus.cc:729] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:45.730302 19933 consensus_queue.cc:260] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:45.730554 19933 ts_tablet_manager.cc:1398] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:45.783751 19940 raft_consensus.cc:482] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.783893 19940 raft_consensus.cc:504] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.784260 19940 leader_election.cc:288] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:45.784644 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a487ec3ad684730b082a278dbf23045" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:45.784644 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a487ec3ad684730b082a278dbf23045" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:45.784791 14957 raft_consensus.cc:2454] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:45.784804 12849 raft_consensus.cc:2454] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:45.785092 12521 leader_election.cc:302] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:45.785251 19940 raft_consensus.cc:2781] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.785315 19940 raft_consensus.cc:482] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.785368 19940 raft_consensus.cc:3037] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.786206 19940 raft_consensus.cc:504] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.786540 19940 leader_election.cc:288] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:45.786898 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a487ec3ad684730b082a278dbf23045" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:45.787037 14957 raft_consensus.cc:3037] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.787019 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a487ec3ad684730b082a278dbf23045" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:45.787148 12849 raft_consensus.cc:3037] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.788107 14957 raft_consensus.cc:2454] T 7a487ec3ad684730b082a278dbf23045 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:45.788210 12849 raft_consensus.cc:2454] T 7a487ec3ad684730b082a278dbf23045 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:45.788499 12523 leader_election.cc:302] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:45.788669 19940 raft_consensus.cc:2781] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.788761 19940 raft_consensus.cc:686] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:45.788897 19940 consensus_queue.cc:237] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:45.792379 12441 catalog_manager.cc:5484] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:45.793344 18656 raft_consensus.cc:482] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:45.793437 18656 raft_consensus.cc:504] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.793777 18656 leader_election.cc:288] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:45.794226 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294cc079a8344f6abe0002063a436e77" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:45.794381 15352 raft_consensus.cc:2454] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:45.794693 12654 leader_election.cc:302] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:45.794809 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294cc079a8344f6abe0002063a436e77" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:45.794934 12979 raft_consensus.cc:2454] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:45.795234 18656 raft_consensus.cc:2781] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:45.795291 18656 raft_consensus.cc:482] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:45.795339 18656 raft_consensus.cc:3037] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.796407 18656 raft_consensus.cc:504] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.796700 18656 leader_election.cc:288] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:45.797065 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294cc079a8344f6abe0002063a436e77" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:45.797068 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294cc079a8344f6abe0002063a436e77" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:45.797178 15352 raft_consensus.cc:3037] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.797178 12979 raft_consensus.cc:3037] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:45.798074 12979 raft_consensus.cc:2454] T 294cc079a8344f6abe0002063a436e77 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:45.798074 15352 raft_consensus.cc:2454] T 294cc079a8344f6abe0002063a436e77 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:45.798339 12654 leader_election.cc:302] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:45.798527 18656 raft_consensus.cc:2781] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:45.798628 18656 raft_consensus.cc:686] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:45.798941 18656 consensus_queue.cc:237] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:45.800871 12441 catalog_manager.cc:5484] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:45.827082 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:46.035094 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:46.040233 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:46.061391 19888 raft_consensus.cc:482] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.061551 19888 raft_consensus.cc:504] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:46.062129 19888 leader_election.cc:288] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:46.062448 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d73d60a1de46cdbcbb15dd46f6bcc0" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:46.062541 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d73d60a1de46cdbcbb15dd46f6bcc0" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:46.062582 14957 raft_consensus.cc:2454] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:46.062681 15221 raft_consensus.cc:2454] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:46.062867 12783 leader_election.cc:302] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:46.063117 19888 raft_consensus.cc:2781] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.063167 19888 raft_consensus.cc:482] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.063206 19888 raft_consensus.cc:3037] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.064056 19888 raft_consensus.cc:504] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:46.064441 19888 leader_election.cc:288] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:46.064754 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d73d60a1de46cdbcbb15dd46f6bcc0" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:46.064847 15221 raft_consensus.cc:3037] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.064857 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d73d60a1de46cdbcbb15dd46f6bcc0" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:46.064981 14957 raft_consensus.cc:3037] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.065560 15221 raft_consensus.cc:2454] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:46.065800 14957 raft_consensus.cc:2454] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:46.065831 12782 leader_election.cc:302] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:46.066078 19888 raft_consensus.cc:2781] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.066256 19888 raft_consensus.cc:686] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:46.066396 19888 consensus_queue.cc:237] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:46.068322 12441 catalog_manager.cc:5484] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:46.083842 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:46.170796 19940 consensus_queue.cc:1035] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.175021 19940 consensus_queue.cc:1035] T 64d71f1f43ce4a918090e94a3679c062 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.187494 19685 consensus_queue.cc:1035] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.187556 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:46.192621 19584 consensus_queue.cc:1035] T 294cc079a8344f6abe0002063a436e77 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.193588 15221 raft_consensus.cc:1261] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.193794 14957 raft_consensus.cc:1261] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.194198 19888 consensus_queue.cc:1035] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.194924 19888 consensus_queue.cc:1035] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.198062 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]). Expected version 1 got 0
I20240501 14:09:46.198405 12441 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]). Expected version 1 got 0
I20240501 14:09:46.200933 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]). Expected version 1 got 0
I20240501 14:09:46.201710 19998 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.201773 12441 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]). Expected version 1 got 0
I20240501 14:09:46.202066 19998 tablet.cc:1620] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:46.199981 19997 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.202574 19997 tablet.cc:1620] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:46.203305 19999 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.203634 19999 tablet.cc:1620] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:46.205632 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e]). Expected version 1 got 0
I20240501 14:09:46.207528 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e] alter complete (version 1)
W20240501 14:09:46.222712 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:46.256490 19940 consensus_queue.cc:1035] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:46.259348 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:46.259985 20001 consensus_queue.cc:1035] T 7a487ec3ad684730b082a278dbf23045 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.309423 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:46.315785 19998 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:46.316046 19997 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:46.316242 19998 tablet.cc:1620] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:46.315892 19999 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:46.316365 19997 tablet.cc:1620] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:46.316805 19999 tablet.cc:1620] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:46.322615 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=6124d86dfb99478ba23f87e49723ff7e] alter complete (version 2)
I20240501 14:09:46.327445 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:46.333046 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90f81e59b24b6493e7e0c870379558 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.333376 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b90f81e59b24b6493e7e0c870379558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.333899 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90f81e59b24b6493e7e0c870379558 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.334093 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 09219251da6341e8a3f58bb3ae4ce413 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.334223 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b90f81e59b24b6493e7e0c870379558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.334414 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09219251da6341e8a3f58bb3ae4ce413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.336135 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 7b90f81e59b24b6493e7e0c870379558 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.336441 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b90f81e59b24b6493e7e0c870379558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.336604 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 09219251da6341e8a3f58bb3ae4ce413 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.336833 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09219251da6341e8a3f58bb3ae4ce413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.336835 20006 tablet_bootstrap.cc:492] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
W20240501 14:09:46.337725 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:46.338148 20005 tablet_bootstrap.cc:492] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:46.338378 20006 tablet_bootstrap.cc:654] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.340261 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 1490b086179d405ca43fb8fd24d90fed (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.341327 20007 tablet_bootstrap.cc:492] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:46.341280 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1490b086179d405ca43fb8fd24d90fed (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.341543 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1490b086179d405ca43fb8fd24d90fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.341499 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 1490b086179d405ca43fb8fd24d90fed (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.341609 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1490b086179d405ca43fb8fd24d90fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.341740 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1490b086179d405ca43fb8fd24d90fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.342478 20007 tablet_bootstrap.cc:654] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.342979 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 09219251da6341e8a3f58bb3ae4ce413 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=c3d153825c0f49be82f24c1171befb1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.343212 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09219251da6341e8a3f58bb3ae4ce413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.343801 20004 tablet_bootstrap.cc:492] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:46.344798 20004 tablet_bootstrap.cc:654] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.344941 20006 tablet_bootstrap.cc:492] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:46.345044 20006 ts_tablet_manager.cc:1367] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:46.345537 20006 raft_consensus.cc:348] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.345677 20006 raft_consensus.cc:374] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.345738 20006 raft_consensus.cc:729] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.345891 20006 consensus_queue.cc:260] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.346441 20006 ts_tablet_manager.cc:1398] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.347342 20007 tablet_bootstrap.cc:492] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:46.347437 20007 ts_tablet_manager.cc:1367] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:46.347920 20007 raft_consensus.cc:348] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.348037 20007 raft_consensus.cc:374] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.348070 20007 raft_consensus.cc:729] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.348184 20007 consensus_queue.cc:260] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.348454 20007 ts_tablet_manager.cc:1398] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.348616 20007 tablet_bootstrap.cc:492] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:46.349534 20007 tablet_bootstrap.cc:654] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.350705 20011 tablet_bootstrap.cc:492] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:46.351653 20005 tablet_bootstrap.cc:654] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.351667 20011 tablet_bootstrap.cc:654] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.352560 20012 tablet_bootstrap.cc:492] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:46.353492 20012 tablet_bootstrap.cc:654] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.353885 20007 tablet_bootstrap.cc:492] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:46.353899 20011 tablet_bootstrap.cc:492] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:46.353986 20011 ts_tablet_manager.cc:1367] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:46.354089 20007 ts_tablet_manager.cc:1367] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:46.354444 20011 raft_consensus.cc:348] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.354595 20011 raft_consensus.cc:374] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.354638 20011 raft_consensus.cc:729] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.354687 20007 raft_consensus.cc:348] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.354816 20007 raft_consensus.cc:374] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.354789 20011 consensus_queue.cc:260] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.354856 20007 raft_consensus.cc:729] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.354967 20007 consensus_queue.cc:260] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.355073 20011 ts_tablet_manager.cc:1398] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.355283 20007 ts_tablet_manager.cc:1398] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.355516 20005 tablet_bootstrap.cc:492] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:46.355616 20005 ts_tablet_manager.cc:1367] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:46.356205 20005 raft_consensus.cc:348] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.356875 20005 raft_consensus.cc:374] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.357858 20005 raft_consensus.cc:729] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.358047 20005 consensus_queue.cc:260] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.358371 20005 ts_tablet_manager.cc:1398] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:46.358985 20012 tablet_bootstrap.cc:492] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:46.359093 20012 ts_tablet_manager.cc:1367] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:46.359581 20012 raft_consensus.cc:348] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.359686 20012 raft_consensus.cc:374] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.359719 20012 raft_consensus.cc:729] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.359848 20012 consensus_queue.cc:260] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.361546 20012 ts_tablet_manager.cc:1398] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.361768 20012 tablet_bootstrap.cc:492] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:46.365438 20012 tablet_bootstrap.cc:654] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.366640 20012 tablet_bootstrap.cc:492] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:46.366799 20012 ts_tablet_manager.cc:1367] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:46.367353 20012 raft_consensus.cc:348] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.367528 20012 raft_consensus.cc:374] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.367604 20012 raft_consensus.cc:729] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.367769 20012 consensus_queue.cc:260] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
W20240501 14:09:46.367959 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:46.368851 20012 ts_tablet_manager.cc:1398] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.369613 20004 tablet_bootstrap.cc:492] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:46.369750 20004 ts_tablet_manager.cc:1367] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:09:46.370384 20004 raft_consensus.cc:348] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.370508 20004 raft_consensus.cc:374] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.370599 20004 raft_consensus.cc:729] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.370743 20004 consensus_queue.cc:260] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.371160 20004 ts_tablet_manager.cc:1398] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.373095 20004 tablet_bootstrap.cc:492] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:46.374148 20004 tablet_bootstrap.cc:654] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.379112 20004 tablet_bootstrap.cc:492] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:46.379261 20004 ts_tablet_manager.cc:1367] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:09:46.379779 20004 raft_consensus.cc:348] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.380815 20004 raft_consensus.cc:374] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.380873 20004 raft_consensus.cc:729] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.381245 20004 consensus_queue.cc:260] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.381824 20004 ts_tablet_manager.cc:1398] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:46.384557 19584 raft_consensus.cc:482] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.384660 19584 raft_consensus.cc:504] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.385022 19584 leader_election.cc:288] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:46.385325 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90f81e59b24b6493e7e0c870379558" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:46.385450 14957 raft_consensus.cc:2454] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:46.385429 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90f81e59b24b6493e7e0c870379558" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:46.385576 12979 raft_consensus.cc:2454] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:46.385744 12652 leader_election.cc:302] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:46.385960 19584 raft_consensus.cc:2781] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.386015 19584 raft_consensus.cc:482] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.386057 19584 raft_consensus.cc:3037] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.387159 19584 raft_consensus.cc:504] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.387492 19584 leader_election.cc:288] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:46.387749 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90f81e59b24b6493e7e0c870379558" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:46.387840 14957 raft_consensus.cc:3037] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.387925 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b90f81e59b24b6493e7e0c870379558" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:46.388053 12979 raft_consensus.cc:3037] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.388883 14957 raft_consensus.cc:2454] T 7b90f81e59b24b6493e7e0c870379558 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:46.389039 12979 raft_consensus.cc:2454] T 7b90f81e59b24b6493e7e0c870379558 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:46.389336 12653 leader_election.cc:302] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:46.389568 19584 raft_consensus.cc:2781] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.389640 19584 raft_consensus.cc:686] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:46.389768 19584 consensus_queue.cc:237] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.392025 12439 catalog_manager.cc:5484] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:46.399439 19810 raft_consensus.cc:482] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.399597 19810 raft_consensus.cc:504] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.399998 19810 leader_election.cc:288] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:46.400353 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09219251da6341e8a3f58bb3ae4ce413" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:46.400489 12717 raft_consensus.cc:2454] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:46.400728 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09219251da6341e8a3f58bb3ae4ce413" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:46.400786 12520 leader_election.cc:302] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:46.400846 15352 raft_consensus.cc:2454] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:46.400928 19810 raft_consensus.cc:2781] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.400978 19810 raft_consensus.cc:482] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.401024 19810 raft_consensus.cc:3037] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.401883 19810 raft_consensus.cc:504] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.402159 19810 leader_election.cc:288] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:46.402434 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09219251da6341e8a3f58bb3ae4ce413" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:46.402529 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09219251da6341e8a3f58bb3ae4ce413" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:46.402570 12717 raft_consensus.cc:3037] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.402631 15352 raft_consensus.cc:3037] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.403563 12717 raft_consensus.cc:2454] T 09219251da6341e8a3f58bb3ae4ce413 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:46.403563 15352 raft_consensus.cc:2454] T 09219251da6341e8a3f58bb3ae4ce413 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:46.403868 12520 leader_election.cc:302] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:46.404099 19810 raft_consensus.cc:2781] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.404201 19810 raft_consensus.cc:686] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:46.404345 19810 consensus_queue.cc:237] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:46.406689 12439 catalog_manager.cc:5484] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:46.426266 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:46.431803 19997 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:46.432155 19999 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:46.432709 19998 tablet.cc:1722] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:46.438249 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=6124d86dfb99478ba23f87e49723ff7e] alter complete (version 3)
I20240501 14:09:46.439868 20003 raft_consensus.cc:482] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.439965 20003 raft_consensus.cc:504] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.440320 20003 leader_election.cc:288] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:46.440726 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1490b086179d405ca43fb8fd24d90fed" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:46.440726 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1490b086179d405ca43fb8fd24d90fed" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:46.440861 14957 raft_consensus.cc:2454] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:46.440869 15352 raft_consensus.cc:2454] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:46.441110 12785 leader_election.cc:302] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:46.441254 20003 raft_consensus.cc:2781] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.441334 20003 raft_consensus.cc:482] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.441404 20003 raft_consensus.cc:3037] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.442296 20003 raft_consensus.cc:504] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.442668 20003 leader_election.cc:288] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:46.442971 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1490b086179d405ca43fb8fd24d90fed" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:46.443080 14957 raft_consensus.cc:3037] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.443075 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1490b086179d405ca43fb8fd24d90fed" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:46.443197 15352 raft_consensus.cc:3037] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.443827 14957 raft_consensus.cc:2454] T 1490b086179d405ca43fb8fd24d90fed P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:46.444064 15352 raft_consensus.cc:2454] T 1490b086179d405ca43fb8fd24d90fed P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:46.444114 12783 leader_election.cc:302] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:46.444283 20003 raft_consensus.cc:2781] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.444355 20003 raft_consensus.cc:686] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:46.444500 20003 consensus_queue.cc:237] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.446542 12438 catalog_manager.cc:5484] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:46.476624 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:46.478114 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:46.540243 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:46.540388 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:46.542248 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 32d73d60a1de46cdbcbb15dd46f6bcc0
I20240501 14:09:46.542943 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:46.542935 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:46.543299 20033 tablet_replica.cc:330] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:46.543287 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:46 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:46.543362 20034 tablet_replica.cc:330] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:46.543457 20033 raft_consensus.cc:2229] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.543571 20034 raft_consensus.cc:2229] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:46.543624 20033 raft_consensus.cc:2258] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.543686 20035 tablet_replica.cc:330] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:46.543850 20035 raft_consensus.cc:2229] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:46.543892 20034 raft_consensus.cc:2258] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.544107 20035 raft_consensus.cc:2258] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:46.544786 20035 ts_tablet_manager.cc:1875] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.545286 20034 ts_tablet_manager.cc:1875] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.545339 20033 ts_tablet_manager.cc:1875] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:46.547410 20035 ts_tablet_manager.cc:1888] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.547518 20035 log.cc:1196] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/32d73d60a1de46cdbcbb15dd46f6bcc0
I20240501 14:09:46.547590 20034 ts_tablet_manager.cc:1888] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.547644 20033 ts_tablet_manager.cc:1888] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:46.547693 20034 log.cc:1196] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/32d73d60a1de46cdbcbb15dd46f6bcc0
I20240501 14:09:46.547757 20033 log.cc:1196] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/32d73d60a1de46cdbcbb15dd46f6bcc0
I20240501 14:09:46.547972 20035 ts_tablet_manager.cc:1909] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:46.548106 20033 ts_tablet_manager.cc:1909] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:46.548134 20034 ts_tablet_manager.cc:1909] T 32d73d60a1de46cdbcbb15dd46f6bcc0 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:46.548822 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table_new_name [id=6124d86dfb99478ba23f87e49723ff7e]) successfully deleted
I20240501 14:09:46.548846 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table_new_name [id=6124d86dfb99478ba23f87e49723ff7e]) successfully deleted
I20240501 14:09:46.549379 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 32d73d60a1de46cdbcbb15dd46f6bcc0 (table rebalancer_dynamic_table_new_name [id=6124d86dfb99478ba23f87e49723ff7e]) successfully deleted
I20240501 14:09:46.643873 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:46.649762 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]), partition=
I20240501 14:09:46.649762 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]), partition=
I20240501 14:09:46.650000 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]), partition=
I20240501 14:09:46.650055 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6ebb6995a34c67a60c5711dfd3e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.650063 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6ebb6995a34c67a60c5711dfd3e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.650218 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f6ebb6995a34c67a60c5711dfd3e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.651980 20006 tablet_bootstrap.cc:492] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:46.652865 20046 tablet_bootstrap.cc:492] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:46.652989 20006 tablet_bootstrap.cc:654] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.653857 20046 tablet_bootstrap.cc:654] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.653914 20005 tablet_bootstrap.cc:492] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:46.654562 20006 tablet_bootstrap.cc:492] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:46.654654 20006 ts_tablet_manager.cc:1367] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:46.654902 20005 tablet_bootstrap.cc:654] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.655151 20006 raft_consensus.cc:348] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.655280 20006 raft_consensus.cc:374] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.655319 20006 raft_consensus.cc:729] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.655448 20006 consensus_queue.cc:260] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.655880 20046 tablet_bootstrap.cc:492] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:46.655959 20046 ts_tablet_manager.cc:1367] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:46.656044 20005 tablet_bootstrap.cc:492] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:46.656132 20005 ts_tablet_manager.cc:1367] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:46.656410 20046 raft_consensus.cc:348] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.656584 20046 raft_consensus.cc:374] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.656634 20046 raft_consensus.cc:729] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.656558 20005 raft_consensus.cc:348] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.656678 20005 raft_consensus.cc:374] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.656734 20005 raft_consensus.cc:729] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.656813 20046 consensus_queue.cc:260] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.656868 20005 consensus_queue.cc:260] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.657147 20005 ts_tablet_manager.cc:1398] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:46.657203 20046 ts_tablet_manager.cc:1398] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.659356 20006 ts_tablet_manager.cc:1398] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20240501 14:09:46.711606 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:46.729530 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:46.779000 19810 raft_consensus.cc:482] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:46.779127 19810 raft_consensus.cc:504] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.779554 19810 leader_election.cc:288] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:46.779904 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6ebb6995a34c67a60c5711dfd3e18b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:46.779938 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6ebb6995a34c67a60c5711dfd3e18b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:46.780038 15221 raft_consensus.cc:2454] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:46.780129 12979 raft_consensus.cc:2454] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:46.780331 12520 leader_election.cc:302] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:46.780566 19810 raft_consensus.cc:2781] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:46.780627 19810 raft_consensus.cc:482] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:46.780676 19810 raft_consensus.cc:3037] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.781553 19810 raft_consensus.cc:504] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.781880 19810 leader_election.cc:288] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:46.782181 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6ebb6995a34c67a60c5711dfd3e18b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:46.782207 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f6ebb6995a34c67a60c5711dfd3e18b" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:46.782302 15221 raft_consensus.cc:3037] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.782322 12979 raft_consensus.cc:3037] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:46.783362 12979 raft_consensus.cc:2454] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:46.783367 15221 raft_consensus.cc:2454] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:46.783658 12520 leader_election.cc:302] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:46.783823 19810 raft_consensus.cc:2781] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:46.783914 19810 raft_consensus.cc:686] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:46.784065 19810 consensus_queue.cc:237] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:46.786100 12443 catalog_manager.cc:5484] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:46.786543 20001 consensus_queue.cc:1035] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.791043 19940 consensus_queue.cc:1035] T 09219251da6341e8a3f58bb3ae4ce413 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.791172 19584 consensus_queue.cc:1035] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.799316 19584 consensus_queue.cc:1035] T 7b90f81e59b24b6493e7e0c870379558 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.908754 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:46.912916 12979 raft_consensus.cc:1261] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.912976 15222 raft_consensus.cc:1261] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:46.913316 19810 consensus_queue.cc:1035] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.913482 19940 consensus_queue.cc:1035] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:46.916693 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]). Expected version 1 got 0
I20240501 14:09:46.916857 19998 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.917151 19998 tablet.cc:1620] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:46.917682 12443 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]). Expected version 1 got 0
I20240501 14:09:46.918298 20071 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.918742 20071 tablet.cc:1620] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:46.919302 12441 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]). Expected version 1 got 0
I20240501 14:09:46.920845 20072 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:46.921140 20072 tablet.cc:1620] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:46.921679 12438 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0]). Expected version 1 got 0
I20240501 14:09:46.923220 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0] alter complete (version 1)
I20240501 14:09:46.963805 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:46.969473 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd65646da3d4d51b6f6971043b92f8a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.969473 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd65646da3d4d51b6f6971043b92f8a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.969786 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd65646da3d4d51b6f6971043b92f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.969786 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd65646da3d4d51b6f6971043b92f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.970366 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 86d02c4bdd80429681c91da65380e2f4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.970366 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab0b777b02a4d2c8cedb58b2c21be27 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.970693 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d02c4bdd80429681c91da65380e2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.970826 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab0b777b02a4d2c8cedb58b2c21be27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.971983 20005 tablet_bootstrap.cc:492] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:46.972235 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 86d02c4bdd80429681c91da65380e2f4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.972496 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d02c4bdd80429681c91da65380e2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.972893 20005 tablet_bootstrap.cc:654] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.973268 20074 tablet_bootstrap.cc:492] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:46.974140 20074 tablet_bootstrap.cc:654] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.974298 20005 tablet_bootstrap.cc:492] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:46.974393 20005 ts_tablet_manager.cc:1367] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:46.974695 20046 tablet_bootstrap.cc:492] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:46.974911 20005 raft_consensus.cc:348] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.975020 20005 raft_consensus.cc:374] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.975064 20005 raft_consensus.cc:729] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.975194 20005 consensus_queue.cc:260] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.975497 20005 ts_tablet_manager.cc:1398] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.975719 20046 tablet_bootstrap.cc:654] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.975854 20074 tablet_bootstrap.cc:492] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:46.975942 20074 ts_tablet_manager.cc:1367] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:46.976543 20074 raft_consensus.cc:348] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.976672 20074 raft_consensus.cc:374] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.976713 20074 raft_consensus.cc:729] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.976850 20074 consensus_queue.cc:260] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.977174 20074 ts_tablet_manager.cc:1398] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:46.977339 20074 tablet_bootstrap.cc:492] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:46.977761 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd65646da3d4d51b6f6971043b92f8a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.978034 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd65646da3d4d51b6f6971043b92f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.978248 20074 tablet_bootstrap.cc:654] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.979084 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab0b777b02a4d2c8cedb58b2c21be27 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.979403 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab0b777b02a4d2c8cedb58b2c21be27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.980469 20074 tablet_bootstrap.cc:492] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:46.980458 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab0b777b02a4d2c8cedb58b2c21be27 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.980561 20074 ts_tablet_manager.cc:1367] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:46.980719 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab0b777b02a4d2c8cedb58b2c21be27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.981006 20074 raft_consensus.cc:348] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:46.981091 20074 raft_consensus.cc:374] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.981117 20074 raft_consensus.cc:729] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.981209 20074 consensus_queue.cc:260] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:46.981448 20080 tablet_bootstrap.cc:492] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:46.981730 20074 ts_tablet_manager.cc:1398] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:46.982770 20080 tablet_bootstrap.cc:654] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.984563 20080 tablet_bootstrap.cc:492] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:46.984777 20080 ts_tablet_manager.cc:1367] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:46.984895 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 86d02c4bdd80429681c91da65380e2f4 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=91d3ae8f33134f2bae5034e9b843ad69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:46.985198 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d02c4bdd80429681c91da65380e2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:46.985519 20080 raft_consensus.cc:348] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.985610 20080 raft_consensus.cc:374] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.985644 20080 raft_consensus.cc:729] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.985821 20080 consensus_queue.cc:260] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:46.986125 20080 ts_tablet_manager.cc:1398] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.988108 20079 tablet_bootstrap.cc:492] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:46.989217 20079 tablet_bootstrap.cc:654] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.989833 20073 tablet_bootstrap.cc:492] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:46.990346 20079 tablet_bootstrap.cc:492] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:46.990479 20079 ts_tablet_manager.cc:1367] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:46.990825 20073 tablet_bootstrap.cc:654] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:46.991082 20079 raft_consensus.cc:348] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.991212 20079 raft_consensus.cc:374] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.991283 20079 raft_consensus.cc:729] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.991441 20079 consensus_queue.cc:260] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.992092 20046 tablet_bootstrap.cc:492] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:46.992201 20046 ts_tablet_manager.cc:1367] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:46.992703 20073 tablet_bootstrap.cc:492] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:46.992818 20073 ts_tablet_manager.cc:1367] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:46.992841 20046 raft_consensus.cc:348] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:46.992916 20079 ts_tablet_manager.cc:1398] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:46.992945 20046 raft_consensus.cc:374] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.992982 20046 raft_consensus.cc:729] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.993093 20079 tablet_bootstrap.cc:492] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:46.993099 20046 consensus_queue.cc:260] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:46.993366 20046 ts_tablet_manager.cc:1398] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:46.993396 20073 raft_consensus.cc:348] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.993500 20073 raft_consensus.cc:374] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:46.993536 20073 raft_consensus.cc:729] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:46.993671 20073 consensus_queue.cc:260] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:46.993976 20073 ts_tablet_manager.cc:1398] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:46.994143 20073 tablet_bootstrap.cc:492] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:46.995580 20073 tablet_bootstrap.cc:654] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
W20240501 14:09:46.998525 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.001150 20079 tablet_bootstrap.cc:654] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.002496 20073 tablet_bootstrap.cc:492] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:47.002542 20079 tablet_bootstrap.cc:492] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:47.002573 20073 ts_tablet_manager.cc:1367] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:47.002635 20079 ts_tablet_manager.cc:1367] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:47.002990 20073 raft_consensus.cc:348] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:47.003085 20073 raft_consensus.cc:374] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.003106 20073 raft_consensus.cc:729] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.003075 20079 raft_consensus.cc:348] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:47.003211 20079 raft_consensus.cc:374] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.003213 20073 consensus_queue.cc:260] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:47.003265 20079 raft_consensus.cc:729] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.003410 20079 consensus_queue.cc:260] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:47.003777 20079 ts_tablet_manager.cc:1398] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.003847 20073 ts_tablet_manager.cc:1398] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.004446 20086 consensus_queue.cc:1035] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.008090 20086 consensus_queue.cc:1035] T 1490b086179d405ca43fb8fd24d90fed P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:47.010787 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:47.027215 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240501 14:09:47.031069 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.032711 20071 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.033042 20071 tablet.cc:1620] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:47.032982 19998 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.033273 19998 tablet.cc:1620] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:47.033701 20072 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:47.034013 20072 tablet.cc:1620] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:47.039312 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=1f7724a0052a4529b07184e40be84ec0] alter complete (version 2)
W20240501 14:09:47.078284 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.090077 19973 raft_consensus.cc:482] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.090199 19973 raft_consensus.cc:504] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:47.090504 19973 leader_election.cc:288] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:47.090840 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd65646da3d4d51b6f6971043b92f8a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:47.090844 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd65646da3d4d51b6f6971043b92f8a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:47.090976 15352 raft_consensus.cc:2454] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:47.090976 12849 raft_consensus.cc:2454] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:47.091255 12916 leader_election.cc:302] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:47.091408 19973 raft_consensus.cc:2781] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.091504 19973 raft_consensus.cc:482] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.091542 19973 raft_consensus.cc:3037] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.092473 19973 raft_consensus.cc:504] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:47.092770 19973 leader_election.cc:288] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:47.093106 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd65646da3d4d51b6f6971043b92f8a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:47.093159 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd65646da3d4d51b6f6971043b92f8a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:47.093221 12849 raft_consensus.cc:3037] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.093298 15352 raft_consensus.cc:3037] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.094094 12849 raft_consensus.cc:2454] T 4bd65646da3d4d51b6f6971043b92f8a P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:47.094130 15352 raft_consensus.cc:2454] T 4bd65646da3d4d51b6f6971043b92f8a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:47.094381 12916 leader_election.cc:302] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:47.094825 19973 raft_consensus.cc:2781] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.094924 19973 raft_consensus.cc:686] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:47.095067 19973 consensus_queue.cc:237] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
W20240501 14:09:47.096362 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.097167 12439 catalog_manager.cc:5484] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:47.139153 19857 raft_consensus.cc:482] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.139287 19857 raft_consensus.cc:504] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:47.139803 19857 leader_election.cc:288] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:47.140154 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab0b777b02a4d2c8cedb58b2c21be27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:47.140179 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab0b777b02a4d2c8cedb58b2c21be27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:47.140300 15351 raft_consensus.cc:2454] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:47.140303 12717 raft_consensus.cc:2454] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:47.140590 14894 leader_election.cc:302] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:47.140750 19857 raft_consensus.cc:2781] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.140817 19857 raft_consensus.cc:482] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.140851 19857 raft_consensus.cc:3037] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.141676 19857 raft_consensus.cc:504] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:47.142006 19857 leader_election.cc:288] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:47.142270 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab0b777b02a4d2c8cedb58b2c21be27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:47.142376 15351 raft_consensus.cc:3037] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.142452 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab0b777b02a4d2c8cedb58b2c21be27" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:47.142549 12717 raft_consensus.cc:3037] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.143378 15351 raft_consensus.cc:2454] T 6ab0b777b02a4d2c8cedb58b2c21be27 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:47.143484 12717 raft_consensus.cc:2454] T 6ab0b777b02a4d2c8cedb58b2c21be27 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:47.143675 14894 leader_election.cc:302] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:47.143908 19857 raft_consensus.cc:2781] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.143976 19857 raft_consensus.cc:686] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:47.144119 19857 consensus_queue.cc:237] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
W20240501 14:09:47.146605 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240501 14:09:47.146489 12439 catalog_manager.cc:5484] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:47.147236 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.147811 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:47.152825 20071 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 14:09:47.153282 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 14:09:47.153293 20072 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 14:09:47.156265 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.156738 19998 tablet.cc:1722] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:47.158574 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=1f7724a0052a4529b07184e40be84ec0] alter complete (version 3)
W20240501 14:09:47.187968 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:09:47.188704 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.238814 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:09:47.245540 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:09:47.246233 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.258599 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.261863 12439 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:47.262045 12439 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:47.264278 12439 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 5f6ebb6995a34c67a60c5711dfd3e18b
I20240501 14:09:47.264892 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6ebb6995a34c67a60c5711dfd3e18b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:47.265038 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6ebb6995a34c67a60c5711dfd3e18b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:47.265236 20112 tablet_replica.cc:330] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:47.265213 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f6ebb6995a34c67a60c5711dfd3e18b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:47 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:47.265353 20113 tablet_replica.cc:330] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:47.265409 20112 raft_consensus.cc:2229] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:47.265532 20114 tablet_replica.cc:330] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:47.265515 20113 raft_consensus.cc:2229] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.265645 20112 raft_consensus.cc:2258] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.265669 20113 raft_consensus.cc:2258] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.265722 20114 raft_consensus.cc:2229] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:47.265882 20114 raft_consensus.cc:2258] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:47.266489 20113 ts_tablet_manager.cc:1875] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.266551 20114 ts_tablet_manager.cc:1875] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.266659 20112 ts_tablet_manager.cc:1875] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:47.268937 20112 ts_tablet_manager.cc:1888] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.269062 20112 log.cc:1196] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f6ebb6995a34c67a60c5711dfd3e18b
I20240501 14:09:47.269073 20113 ts_tablet_manager.cc:1888] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.269181 20113 log.cc:1196] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f6ebb6995a34c67a60c5711dfd3e18b
I20240501 14:09:47.269467 20112 ts_tablet_manager.cc:1909] T 5f6ebb6995a34c67a60c5711dfd3e18b P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:47.269551 20113 ts_tablet_manager.cc:1909] T 5f6ebb6995a34c67a60c5711dfd3e18b P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:47.269603 20114 ts_tablet_manager.cc:1888] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:47.269685 20114 log.cc:1196] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/5f6ebb6995a34c67a60c5711dfd3e18b
I20240501 14:09:47.269959 20114 ts_tablet_manager.cc:1909] T 5f6ebb6995a34c67a60c5711dfd3e18b P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:47.270304 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table_new_name [id=1f7724a0052a4529b07184e40be84ec0]) successfully deleted
I20240501 14:09:47.270524 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table_new_name [id=1f7724a0052a4529b07184e40be84ec0]) successfully deleted
I20240501 14:09:47.270826 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 5f6ebb6995a34c67a60c5711dfd3e18b (table rebalancer_dynamic_table_new_name [id=1f7724a0052a4529b07184e40be84ec0]) successfully deleted
W20240501 14:09:47.279800 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.286448 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.309582 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240501 14:09:47.310153 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:09:47.327987 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 14:09:47.328649 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.332024 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:09:47.366362 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:47.372648 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]), partition=
I20240501 14:09:47.372957 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfbc1a72bf14d5abf356ebc786feef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.372923 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]), partition=
I20240501 14:09:47.373078 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]), partition=
I20240501 14:09:47.373211 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfbc1a72bf14d5abf356ebc786feef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.373278 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfbc1a72bf14d5abf356ebc786feef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.374974 20080 tablet_bootstrap.cc:492] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:47.375802 20115 tablet_bootstrap.cc:492] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:47.375828 20046 tablet_bootstrap.cc:492] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:47.375900 20080 tablet_bootstrap.cc:654] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.376727 20115 tablet_bootstrap.cc:654] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.376722 20046 tablet_bootstrap.cc:654] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.377688 20080 tablet_bootstrap.cc:492] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:47.377777 20080 ts_tablet_manager.cc:1367] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:47.378216 20080 raft_consensus.cc:348] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.378322 20080 raft_consensus.cc:374] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.378348 20080 raft_consensus.cc:729] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.378453 20080 consensus_queue.cc:260] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.378769 20080 ts_tablet_manager.cc:1398] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.380448 20046 tablet_bootstrap.cc:492] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:47.380549 20046 ts_tablet_manager.cc:1367] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:47.381013 20046 raft_consensus.cc:348] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.381119 20046 raft_consensus.cc:374] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.381150 20046 raft_consensus.cc:729] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.381254 20046 consensus_queue.cc:260] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.381534 20046 ts_tablet_manager.cc:1398] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.383780 20115 tablet_bootstrap.cc:492] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:47.384074 20115 ts_tablet_manager.cc:1367] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:47.384661 20115 raft_consensus.cc:348] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.384783 20115 raft_consensus.cc:374] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.384831 20115 raft_consensus.cc:729] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.384975 20115 consensus_queue.cc:260] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.385288 20115 ts_tablet_manager.cc:1398] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.409224 19823 raft_consensus.cc:482] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.409389 19823 raft_consensus.cc:504] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:47.409806 19823 leader_election.cc:288] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:47.410230 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d02c4bdd80429681c91da65380e2f4" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:47.410346 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d02c4bdd80429681c91da65380e2f4" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:47.410385 12849 raft_consensus.cc:2454] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:47.410483 12717 raft_consensus.cc:2454] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:47.410703 15157 leader_election.cc:302] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:47.410914 19823 raft_consensus.cc:2781] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.410977 19823 raft_consensus.cc:482] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.411026 19823 raft_consensus.cc:3037] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.411984 19823 raft_consensus.cc:504] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:47.412297 19823 leader_election.cc:288] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:47.412581 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d02c4bdd80429681c91da65380e2f4" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:47.412711 12717 raft_consensus.cc:3037] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.413843 12717 raft_consensus.cc:2454] T 86d02c4bdd80429681c91da65380e2f4 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:47.414098 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d02c4bdd80429681c91da65380e2f4" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:47.414207 12849 raft_consensus.cc:3037] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.414991 12849 raft_consensus.cc:2454] T 86d02c4bdd80429681c91da65380e2f4 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:47.415285 15157 leader_election.cc:302] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:47.415498 19823 raft_consensus.cc:2781] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.415637 19823 raft_consensus.cc:686] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:47.415777 19823 consensus_queue.cc:237] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:47.417879 12439 catalog_manager.cc:5484] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:47.437995 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.444607 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:47.446882 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:09:47.572952 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.599417 19857 consensus_queue.cc:1035] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:47.599725 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:47.603471 19857 consensus_queue.cc:1035] T 6ab0b777b02a4d2c8cedb58b2c21be27 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.639962 19973 consensus_queue.cc:1035] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.661339 19973 consensus_queue.cc:1035] T 4bd65646da3d4d51b6f6971043b92f8a P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.804836 19823 raft_consensus.cc:482] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:47.804987 19823 raft_consensus.cc:504] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.805393 19823 leader_election.cc:288] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:47.805794 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfbc1a72bf14d5abf356ebc786feef6" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:47.805934 14957 raft_consensus.cc:2454] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:47.805938 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfbc1a72bf14d5abf356ebc786feef6" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:47.806053 12586 raft_consensus.cc:2454] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:47.806325 15155 leader_election.cc:302] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:47.806596 19823 raft_consensus.cc:2781] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:47.806650 19823 raft_consensus.cc:482] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:47.806689 19823 raft_consensus.cc:3037] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.807825 19823 raft_consensus.cc:504] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.808449 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfbc1a72bf14d5abf356ebc786feef6" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:47.808560 14957 raft_consensus.cc:3037] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.809275 14957 raft_consensus.cc:2454] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:47.809517 15155 leader_election.cc:302] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:47.809675 20120 raft_consensus.cc:2781] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:47.809823 20120 raft_consensus.cc:686] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:47.809979 20120 consensus_queue.cc:237] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.812144 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfbc1a72bf14d5abf356ebc786feef6" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:47.812256 12586 raft_consensus.cc:3037] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:47.812297 12439 catalog_manager.cc:5484] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:47.813205 12586 raft_consensus.cc:2454] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:47.814893 19823 leader_election.cc:288] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:47.889513 20120 consensus_queue.cc:1035] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.892691 20120 consensus_queue.cc:1035] T 86d02c4bdd80429681c91da65380e2f4 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.926250 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:47.930490 12586 raft_consensus.cc:1261] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.930569 14958 raft_consensus.cc:1261] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:47.930961 19823 consensus_queue.cc:1035] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.931102 20120 consensus_queue.cc:1035] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:47.934316 20160 mvcc.cc:205] Tried to move back new op lower bound from 7022889320159731712 to 7022889319670710272. Current Snapshot: MvccSnapshot[applied={T|T < 7022889320159731712}]
I20240501 14:09:47.934624 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]). Expected version 1 got 0
I20240501 14:09:47.935148 12439 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]). Expected version 1 got 0
I20240501 14:09:47.935669 20170 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:47.936779 20170 tablet.cc:1620] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:47.936781 20171 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:47.937134 20171 tablet.cc:1620] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:47.937328 20172 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:47.937691 20172 tablet.cc:1620] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:47.938099 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]). Expected version 1 got 0
W20240501 14:09:47.939286 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 14:09:47.941395 12439 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1]). Expected version 1 got 0
W20240501 14:09:47.941737 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:47.942855 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1] alter complete (version 1)
W20240501 14:09:47.943061 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:47.957700 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:47.964253 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 845092e1d3cd496cb7f80b508a3874b0 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.964352 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac348a02f2f44d9a6bb81a98609d1bb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.964603 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845092e1d3cd496cb7f80b508a3874b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.964659 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac348a02f2f44d9a6bb81a98609d1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.964712 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 845092e1d3cd496cb7f80b508a3874b0 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.964781 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 6f43ec9540474daba26f916ef89bb200 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.964989 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845092e1d3cd496cb7f80b508a3874b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.965399 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f43ec9540474daba26f916ef89bb200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.967746 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac348a02f2f44d9a6bb81a98609d1bb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.967947 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 6f43ec9540474daba26f916ef89bb200 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.968060 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac348a02f2f44d9a6bb81a98609d1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.968240 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f43ec9540474daba26f916ef89bb200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.968617 20175 tablet_bootstrap.cc:492] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:47.968781 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 845092e1d3cd496cb7f80b508a3874b0 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.969022 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 845092e1d3cd496cb7f80b508a3874b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.969611 20175 tablet_bootstrap.cc:654] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.970774 20179 tablet_bootstrap.cc:492] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:47.970906 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac348a02f2f44d9a6bb81a98609d1bb (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.971144 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac348a02f2f44d9a6bb81a98609d1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.971427 20180 tablet_bootstrap.cc:492] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:47.972029 20179 tablet_bootstrap.cc:654] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.972292 20181 tablet_bootstrap.cc:492] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:47.972512 20180 tablet_bootstrap.cc:654] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.973506 20181 tablet_bootstrap.cc:654] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.973644 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 6f43ec9540474daba26f916ef89bb200 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=b424a08f60f645b78b7f7f3d14c92c88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:47.973901 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f43ec9540474daba26f916ef89bb200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:47.974520 20176 tablet_bootstrap.cc:492] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:47.975555 20176 tablet_bootstrap.cc:654] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.975675 20180 tablet_bootstrap.cc:492] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:47.975776 20180 ts_tablet_manager.cc:1367] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:47.976279 20180 raft_consensus.cc:348] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:47.976429 20180 raft_consensus.cc:374] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.967464 20177 tablet_bootstrap.cc:492] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:47.976495 20180 raft_consensus.cc:729] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.976619 20180 consensus_queue.cc:260] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:47.976776 20176 tablet_bootstrap.cc:492] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:47.977027 20180 ts_tablet_manager.cc:1398] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.976891 20176 ts_tablet_manager.cc:1367] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:47.977464 20177 tablet_bootstrap.cc:654] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.977526 20176 raft_consensus.cc:348] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:47.977640 20176 raft_consensus.cc:374] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.977710 20176 raft_consensus.cc:729] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.977849 20176 consensus_queue.cc:260] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:47.978332 20176 ts_tablet_manager.cc:1398] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.979360 20177 tablet_bootstrap.cc:492] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:47.979437 20181 tablet_bootstrap.cc:492] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:47.979470 20177 ts_tablet_manager.cc:1367] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:47.979544 20181 ts_tablet_manager.cc:1367] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:47.979908 20181 raft_consensus.cc:348] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.979904 20177 raft_consensus.cc:348] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.979988 20181 raft_consensus.cc:374] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.980008 20181 raft_consensus.cc:729] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.980011 20177 raft_consensus.cc:374] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.980054 20177 raft_consensus.cc:729] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.980096 20181 consensus_queue.cc:260] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.980190 20177 consensus_queue.cc:260] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.980340 20181 ts_tablet_manager.cc:1398] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.980487 20177 ts_tablet_manager.cc:1398] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.980521 20181 tablet_bootstrap.cc:492] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:47.980634 20177 tablet_bootstrap.cc:492] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:47.981501 20177 tablet_bootstrap.cc:654] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.982008 20181 tablet_bootstrap.cc:654] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.982934 20179 tablet_bootstrap.cc:492] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:47.983059 20179 ts_tablet_manager.cc:1367] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:47.983542 20179 raft_consensus.cc:348] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:47.983628 20179 raft_consensus.cc:374] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.983647 20179 raft_consensus.cc:729] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.983772 20179 consensus_queue.cc:260] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:47.984019 20176 tablet_bootstrap.cc:492] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:47.984091 20179 ts_tablet_manager.cc:1398] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.985232 20176 tablet_bootstrap.cc:654] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:47.986691 20181 tablet_bootstrap.cc:492] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:47.986819 20181 ts_tablet_manager.cc:1367] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:47.987320 20181 raft_consensus.cc:348] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:47.987432 20181 raft_consensus.cc:374] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.987466 20181 raft_consensus.cc:729] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.987608 20181 consensus_queue.cc:260] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:47.987896 20181 ts_tablet_manager.cc:1398] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:47.989086 20177 tablet_bootstrap.cc:492] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:47.989215 20177 ts_tablet_manager.cc:1367] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:47.989727 20177 raft_consensus.cc:348] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:47.989848 20177 raft_consensus.cc:374] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.989893 20177 raft_consensus.cc:729] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.990026 20177 consensus_queue.cc:260] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:47.990340 20177 ts_tablet_manager.cc:1398] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:47.991196 20176 tablet_bootstrap.cc:492] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:47.991289 20176 ts_tablet_manager.cc:1367] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:47.991782 20176 raft_consensus.cc:348] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:47.991961 20176 raft_consensus.cc:374] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.992058 20176 raft_consensus.cc:729] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.992425 20176 consensus_queue.cc:260] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:47.992802 20176 ts_tablet_manager.cc:1398] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:47.999121 20175 tablet_bootstrap.cc:492] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:47.999225 20175 ts_tablet_manager.cc:1367] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240501 14:09:47.999630 20175 raft_consensus.cc:348] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:47.999728 20175 raft_consensus.cc:374] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:47.999766 20175 raft_consensus.cc:729] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:47.999898 20175 consensus_queue.cc:260] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.000169 20175 ts_tablet_manager.cc:1398] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.046973 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:48.052312 19584 raft_consensus.cc:482] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.052425 19584 raft_consensus.cc:504] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.052798 19584 leader_election.cc:288] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:48.053159 20170 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.053213 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f43ec9540474daba26f916ef89bb200" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:48.053344 12979 raft_consensus.cc:2454] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:48.053508 20170 tablet.cc:1620] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:48.053714 12653 leader_election.cc:302] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:48.053953 19584 raft_consensus.cc:2781] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.054008 19584 raft_consensus.cc:482] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.054040 19584 raft_consensus.cc:3037] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.054085 20171 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.054425 20171 tablet.cc:1620] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:48.054736 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f43ec9540474daba26f916ef89bb200" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:48.054896 14957 raft_consensus.cc:2454] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:48.055115 19584 raft_consensus.cc:504] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.055433 19584 leader_election.cc:288] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:48.055996 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f43ec9540474daba26f916ef89bb200" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:48.056098 12979 raft_consensus.cc:3037] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.057066 12979 raft_consensus.cc:2454] T 6f43ec9540474daba26f916ef89bb200 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:48.057410 12653 leader_election.cc:302] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:48.057415 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f43ec9540474daba26f916ef89bb200" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:48.057536 14957 raft_consensus.cc:3037] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.057736 19685 raft_consensus.cc:2781] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.057808 19685 raft_consensus.cc:686] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:48.057929 19685 consensus_queue.cc:237] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.058192 20172 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.058552 14957 raft_consensus.cc:2454] T 6f43ec9540474daba26f916ef89bb200 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:48.058547 20172 tablet.cc:1620] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:48.062065 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=5d3b463f7af140cea6e13905bb9e9bc1] alter complete (version 2)
I20240501 14:09:48.063109 12441 catalog_manager.cc:5484] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:48.063570 19810 raft_consensus.cc:482] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.063666 19810 raft_consensus.cc:504] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.064065 19810 leader_election.cc:288] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:48.064620 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845092e1d3cd496cb7f80b508a3874b0" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:48.064743 15221 raft_consensus.cc:2454] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:48.065011 12520 leader_election.cc:302] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:48.065156 19810 raft_consensus.cc:2781] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.065193 19810 raft_consensus.cc:482] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.065224 19810 raft_consensus.cc:3037] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.066121 19810 raft_consensus.cc:504] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.066345 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845092e1d3cd496cb7f80b508a3874b0" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:48.066486 19810 leader_election.cc:288] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:48.066474 12979 raft_consensus.cc:2454] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:48.066862 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845092e1d3cd496cb7f80b508a3874b0" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:48.066991 15221 raft_consensus.cc:3037] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.066983 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "845092e1d3cd496cb7f80b508a3874b0" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:48.067097 12979 raft_consensus.cc:3037] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.067786 15221 raft_consensus.cc:2454] T 845092e1d3cd496cb7f80b508a3874b0 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:48.067930 12979 raft_consensus.cc:2454] T 845092e1d3cd496cb7f80b508a3874b0 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:48.068048 12520 leader_election.cc:302] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:48.068336 19810 raft_consensus.cc:2781] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.068445 19810 raft_consensus.cc:686] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:48.068596 19810 consensus_queue.cc:237] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.070922 12441 catalog_manager.cc:5484] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:48.168078 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:48.173236 20170 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.173992 20172 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.175205 20171 tablet.cc:1722] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.179188 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=5d3b463f7af140cea6e13905bb9e9bc1] alter complete (version 3)
I20240501 14:09:48.244671 20009 raft_consensus.cc:482] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.244783 20009 raft_consensus.cc:504] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:48.245261 20009 leader_election.cc:288] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:48.245556 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac348a02f2f44d9a6bb81a98609d1bb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:48.245612 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac348a02f2f44d9a6bb81a98609d1bb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:48.245702 12586 raft_consensus.cc:2454] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:48.245762 12849 raft_consensus.cc:2454] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:48.246068 14894 leader_election.cc:302] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:48.246384 20009 raft_consensus.cc:2781] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.246455 20009 raft_consensus.cc:482] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.246496 20009 raft_consensus.cc:3037] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.247923 20009 raft_consensus.cc:504] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:48.248294 20009 leader_election.cc:288] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:48.248613 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac348a02f2f44d9a6bb81a98609d1bb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:48.248732 12586 raft_consensus.cc:3037] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.248829 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac348a02f2f44d9a6bb81a98609d1bb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:48.248947 12849 raft_consensus.cc:3037] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.249768 12586 raft_consensus.cc:2454] T 1ac348a02f2f44d9a6bb81a98609d1bb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:48.249975 12849 raft_consensus.cc:2454] T 1ac348a02f2f44d9a6bb81a98609d1bb P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:48.250065 14893 leader_election.cc:302] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:48.250257 20009 raft_consensus.cc:2781] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.250355 20009 raft_consensus.cc:686] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:48.250483 20009 consensus_queue.cc:237] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:48.252827 12439 catalog_manager.cc:5484] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:48.282791 12439 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:48.282959 12439 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:48.284942 12439 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 2dfbc1a72bf14d5abf356ebc786feef6
I20240501 14:09:48.285565 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfbc1a72bf14d5abf356ebc786feef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:48.285604 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfbc1a72bf14d5abf356ebc786feef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:48.285704 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfbc1a72bf14d5abf356ebc786feef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:48.286077 20206 tablet_replica.cc:330] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:48.286129 20207 tablet_replica.cc:330] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:48.286267 20205 tablet_replica.cc:330] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:48.286267 20206 raft_consensus.cc:2229] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.286348 20207 raft_consensus.cc:2229] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:48.286409 20205 raft_consensus.cc:2229] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.286434 20206 raft_consensus.cc:2258] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.286528 20205 raft_consensus.cc:2258] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.286578 20207 raft_consensus.cc:2258] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.287189 20205 ts_tablet_manager.cc:1875] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.287189 20207 ts_tablet_manager.cc:1875] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.289412 20205 ts_tablet_manager.cc:1888] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.289494 20205 log.cc:1196] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/2dfbc1a72bf14d5abf356ebc786feef6
I20240501 14:09:48.289495 20207 ts_tablet_manager.cc:1888] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.289664 20206 ts_tablet_manager.cc:1875] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.289610 20207 log.cc:1196] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/2dfbc1a72bf14d5abf356ebc786feef6
I20240501 14:09:48.289952 20205 ts_tablet_manager.cc:1909] T 2dfbc1a72bf14d5abf356ebc786feef6 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:48.290099 20207 ts_tablet_manager.cc:1909] T 2dfbc1a72bf14d5abf356ebc786feef6 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:48.290630 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table_new_name [id=5d3b463f7af140cea6e13905bb9e9bc1]) successfully deleted
I20240501 14:09:48.290843 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table_new_name [id=5d3b463f7af140cea6e13905bb9e9bc1]) successfully deleted
I20240501 14:09:48.291985 20206 ts_tablet_manager.cc:1888] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.292073 20206 log.cc:1196] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/2dfbc1a72bf14d5abf356ebc786feef6
I20240501 14:09:48.292315 20206 ts_tablet_manager.cc:1909] T 2dfbc1a72bf14d5abf356ebc786feef6 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:48.292912 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 2dfbc1a72bf14d5abf356ebc786feef6 (table rebalancer_dynamic_table_new_name [id=5d3b463f7af140cea6e13905bb9e9bc1]) successfully deleted
W20240501 14:09:48.328384 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.387039 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:48.392566 15332 tablet_service.cc:1467] Processing CreateTablet for tablet b88e8ea21db9422b9b54bdc0ad609fbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]), partition=
I20240501 14:09:48.392555 12958 tablet_service.cc:1467] Processing CreateTablet for tablet b88e8ea21db9422b9b54bdc0ad609fbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]), partition=
I20240501 14:09:48.392832 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88e8ea21db9422b9b54bdc0ad609fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.392832 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88e8ea21db9422b9b54bdc0ad609fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.395188 20218 tablet_bootstrap.cc:492] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:48.395188 20177 tablet_bootstrap.cc:492] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:48.396219 20177 tablet_bootstrap.cc:654] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.396219 20218 tablet_bootstrap.cc:654] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.397862 20218 tablet_bootstrap.cc:492] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:48.397979 20218 ts_tablet_manager.cc:1367] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:48.398520 20218 raft_consensus.cc:348] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.398803 20218 raft_consensus.cc:374] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.398870 20218 raft_consensus.cc:729] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.399003 20218 consensus_queue.cc:260] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.399338 20218 ts_tablet_manager.cc:1398] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.402043 12827 tablet_service.cc:1467] Processing CreateTablet for tablet b88e8ea21db9422b9b54bdc0ad609fbe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]), partition=
I20240501 14:09:48.402310 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88e8ea21db9422b9b54bdc0ad609fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.402441 20177 tablet_bootstrap.cc:492] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:48.402542 20177 ts_tablet_manager.cc:1367] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:48.403088 20177 raft_consensus.cc:348] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.403203 20177 raft_consensus.cc:374] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.403236 20177 raft_consensus.cc:729] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.404887 20179 tablet_bootstrap.cc:492] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:48.403374 20177 consensus_queue.cc:260] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.405788 20177 ts_tablet_manager.cc:1398] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:48.405836 20179 tablet_bootstrap.cc:654] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.407095 20179 tablet_bootstrap.cc:492] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:48.407178 20179 ts_tablet_manager.cc:1367] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:48.407580 20179 raft_consensus.cc:348] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.407670 20179 raft_consensus.cc:374] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.407697 20179 raft_consensus.cc:729] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.407814 20179 consensus_queue.cc:260] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.408097 20179 ts_tablet_manager.cc:1398] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 14:09:48.410037 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:48.433367 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.465523 19971 raft_consensus.cc:482] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.465664 19971 raft_consensus.cc:504] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.466084 19971 leader_election.cc:288] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:48.466389 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88e8ea21db9422b9b54bdc0ad609fbe" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:48.466454 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88e8ea21db9422b9b54bdc0ad609fbe" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:48.466537 15352 raft_consensus.cc:2454] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:48.466579 12979 raft_consensus.cc:2454] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:48.466820 12785 leader_election.cc:302] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:48.467011 19971 raft_consensus.cc:2781] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.467075 19971 raft_consensus.cc:482] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.467125 19971 raft_consensus.cc:3037] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.468096 19971 raft_consensus.cc:504] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.468477 19971 leader_election.cc:288] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:48.468778 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88e8ea21db9422b9b54bdc0ad609fbe" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:48.468786 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88e8ea21db9422b9b54bdc0ad609fbe" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:48.468891 15352 raft_consensus.cc:3037] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.468891 12979 raft_consensus.cc:3037] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.469798 12979 raft_consensus.cc:2454] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:48.469798 15352 raft_consensus.cc:2454] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:48.470086 12785 leader_election.cc:302] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:48.470232 19971 raft_consensus.cc:2781] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.470373 19971 raft_consensus.cc:686] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:48.470479 19971 consensus_queue.cc:237] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:48.472594 12439 catalog_manager.cc:5484] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:48.475198 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.515286 19685 consensus_queue.cc:1035] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.519631 19584 consensus_queue.cc:1035] T 6f43ec9540474daba26f916ef89bb200 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.550460 19810 consensus_queue.cc:1035] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.554497 20240 consensus_queue.cc:1035] T 845092e1d3cd496cb7f80b508a3874b0 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:48.578958 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.582551 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:48.586855 12979 raft_consensus.cc:1261] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:48.586858 15352 raft_consensus.cc:1261] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:48.587234 19971 consensus_queue.cc:1035] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.587409 20222 consensus_queue.cc:1035] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.590029 12439 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]). Expected version 1 got 0
I20240501 14:09:48.591512 12440 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]). Expected version 1 got 0
I20240501 14:09:48.592235 12441 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]). Expected version 1 got 0
I20240501 14:09:48.592224 20247 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:48.592541 20246 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:48.592993 12440 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]). Expected version 1 got 0
I20240501 14:09:48.593070 20246 tablet.cc:1620] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:48.593855 20247 tablet.cc:1620] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:48.598837 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e] alter complete (version 1)
I20240501 14:09:48.599058 20248 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:48.599359 20248 tablet.cc:1620] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:48.600510 12440 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e]). Expected version 1 got 0
I20240501 14:09:48.703039 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:48.708637 20247 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.708218 20246 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.708958 20247 tablet.cc:1620] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:48.709090 20246 tablet.cc:1620] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:48.709277 20248 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:48.709591 20248 tablet.cc:1620] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:48.714228 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=40edc73f1c0e4417a63ffa5dd000e59e] alter complete (version 2)
I20240501 14:09:48.725335 20009 consensus_queue.cc:1035] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:48.728089 20009 consensus_queue.cc:1035] T 1ac348a02f2f44d9a6bb81a98609d1bb P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:48.769852 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.772511 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:48.778390 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca1be8b60de430f8231f9868edfab6e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.778697 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab213d51c1440819360a63e12aad705 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.778788 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca1be8b60de430f8231f9868edfab6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.778978 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca1be8b60de430f8231f9868edfab6e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.779557 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 650c3f2054da4995a3a1668ecae11836 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.779765 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650c3f2054da4995a3a1668ecae11836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.781105 20262 tablet_bootstrap.cc:492] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:48.781077 15200 tablet_service.cc:1467] Processing CreateTablet for tablet 650c3f2054da4995a3a1668ecae11836 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.781294 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca1be8b60de430f8231f9868edfab6e (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.781502 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca1be8b60de430f8231f9868edfab6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.781558 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab213d51c1440819360a63e12aad705 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.781793 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab213d51c1440819360a63e12aad705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.781996 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab213d51c1440819360a63e12aad705 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.782245 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab213d51c1440819360a63e12aad705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.782416 20262 tablet_bootstrap.cc:654] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.783109 20177 tablet_bootstrap.cc:492] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:48.783599 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 650c3f2054da4995a3a1668ecae11836 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=3e78c83d83374e54b4d7623bcb13e5cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:48.783815 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650c3f2054da4995a3a1668ecae11836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.783895 20177 tablet_bootstrap.cc:654] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.783903 20218 tablet_bootstrap.cc:492] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:48.784857 20218 tablet_bootstrap.cc:654] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.778987 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab213d51c1440819360a63e12aad705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.785761 15200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650c3f2054da4995a3a1668ecae11836. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.786181 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca1be8b60de430f8231f9868edfab6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:48.786938 20266 tablet_bootstrap.cc:492] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:48.787803 20266 tablet_bootstrap.cc:654] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.787927 20177 tablet_bootstrap.cc:492] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:48.788025 20177 ts_tablet_manager.cc:1367] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:48.788467 20177 raft_consensus.cc:348] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.788570 20177 raft_consensus.cc:374] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.788599 20177 raft_consensus.cc:729] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.788703 20177 consensus_queue.cc:260] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.788877 20266 tablet_bootstrap.cc:492] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:48.788926 20177 ts_tablet_manager.cc:1398] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.788954 20266 ts_tablet_manager.cc:1367] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:48.789093 20177 tablet_bootstrap.cc:492] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:48.789379 20266 raft_consensus.cc:348] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.789475 20266 raft_consensus.cc:374] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.789505 20266 raft_consensus.cc:729] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.789637 20266 consensus_queue.cc:260] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.789969 20266 ts_tablet_manager.cc:1398] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.790606 20177 tablet_bootstrap.cc:654] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.791276 20267 tablet_bootstrap.cc:492] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:48.791719 20218 tablet_bootstrap.cc:492] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:48.791831 20218 ts_tablet_manager.cc:1367] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:48.792328 20267 tablet_bootstrap.cc:654] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.792387 20218 raft_consensus.cc:348] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.792487 20218 raft_consensus.cc:374] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.792531 20218 raft_consensus.cc:729] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.792652 20218 consensus_queue.cc:260] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.792948 20218 ts_tablet_manager.cc:1398] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.793386 20267 tablet_bootstrap.cc:492] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:48.793467 20267 ts_tablet_manager.cc:1367] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:48.793963 20267 raft_consensus.cc:348] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.794065 20267 raft_consensus.cc:374] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.794093 20267 raft_consensus.cc:729] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.794201 20267 consensus_queue.cc:260] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.794495 20267 ts_tablet_manager.cc:1398] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.795358 20267 tablet_bootstrap.cc:492] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:48.796284 20267 tablet_bootstrap.cc:654] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.797844 20267 tablet_bootstrap.cc:492] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:48.797998 20267 ts_tablet_manager.cc:1367] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:48.798532 20267 raft_consensus.cc:348] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.798707 20267 raft_consensus.cc:374] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.798802 20267 raft_consensus.cc:729] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.799511 20177 tablet_bootstrap.cc:492] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:48.799597 20177 ts_tablet_manager.cc:1367] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:09:48.799633 20262 tablet_bootstrap.cc:492] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:48.799731 20262 ts_tablet_manager.cc:1367] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:48.799682 20267 consensus_queue.cc:260] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.799971 20267 ts_tablet_manager.cc:1398] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:48.800055 20177 raft_consensus.cc:348] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.800124 20267 tablet_bootstrap.cc:492] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:48.800148 20177 raft_consensus.cc:374] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.800176 20177 raft_consensus.cc:729] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.800149 20262 raft_consensus.cc:348] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.800292 20262 raft_consensus.cc:374] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.800329 20262 raft_consensus.cc:729] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.800297 20177 consensus_queue.cc:260] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.800449 20262 consensus_queue.cc:260] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.800570 20177 ts_tablet_manager.cc:1398] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.800694 20262 ts_tablet_manager.cc:1398] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:48.800850 20262 tablet_bootstrap.cc:492] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:48.801009 20203 raft_consensus.cc:482] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.801079 20203 raft_consensus.cc:504] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.801514 20203 leader_election.cc:288] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:48.801692 20262 tablet_bootstrap.cc:654] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.801896 20267 tablet_bootstrap.cc:654] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:48.802459 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c3f2054da4995a3a1668ecae11836" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:48.802560 15221 raft_consensus.cc:2454] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:48.803043 12913 leader_election.cc:302] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:48.803220 20203 raft_consensus.cc:2781] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.803267 20203 raft_consensus.cc:482] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.803313 20203 raft_consensus.cc:3037] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.804395 20203 raft_consensus.cc:504] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.804724 20203 leader_election.cc:288] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:48.805500 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c3f2054da4995a3a1668ecae11836" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:48.805758 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c3f2054da4995a3a1668ecae11836" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:48.807399 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c3f2054da4995a3a1668ecae11836" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:48.807503 15221 raft_consensus.cc:3037] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.808257 15221 raft_consensus.cc:2454] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:48.808507 12913 leader_election.cc:302] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:48.808650 19736 raft_consensus.cc:2781] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.808708 19736 raft_consensus.cc:686] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:48.808816 19736 consensus_queue.cc:237] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
W20240501 14:09:48.808844 12915 leader_election.cc:341] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:09:48.809060 12915 leader_election.cc:341] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:09:48.811028 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.813380 12439 catalog_manager.cc:5484] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:48.814301 20262 tablet_bootstrap.cc:492] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:48.814388 20262 ts_tablet_manager.cc:1367] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:48.814836 20262 raft_consensus.cc:348] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.814929 20262 raft_consensus.cc:374] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.814968 20262 raft_consensus.cc:729] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.815366 20267 tablet_bootstrap.cc:492] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:48.815336 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:48.815356 20262 consensus_queue.cc:260] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.815475 20267 ts_tablet_manager.cc:1367] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240501 14:09:48.815866 20262 ts_tablet_manager.cc:1398] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:48.815939 20267 raft_consensus.cc:348] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.816061 20267 raft_consensus.cc:374] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:48.816097 20267 raft_consensus.cc:729] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:48.816216 20267 consensus_queue.cc:260] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.816498 20267 ts_tablet_manager.cc:1398] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:48.822085 20246 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.822537 20248 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.822537 20247 tablet.cc:1722] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:48.829285 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=40edc73f1c0e4417a63ffa5dd000e59e] alter complete (version 3)
W20240501 14:09:48.850080 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:48.852557 20240 raft_consensus.cc:482] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.852666 20240 raft_consensus.cc:504] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:48.853171 20240 leader_election.cc:288] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:48.853479 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c3f2054da4995a3a1668ecae11836" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:48.853479 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650c3f2054da4995a3a1668ecae11836" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:48.853660 15221 raft_consensus.cc:2379] T 650c3f2054da4995a3a1668ecae11836 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a3cc0660eed427eaee38164a8c1d4c3 in current term 1: Already voted for candidate a6e94562bee149c2ace25672f0f9bf50 in this term.
I20240501 14:09:48.853937 12520 leader_election.cc:302] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3; no voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50
I20240501 14:09:48.854104 20240 raft_consensus.cc:3037] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.855105 20240 raft_consensus.cc:2726] T 650c3f2054da4995a3a1668ecae11836 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:48.879315 20137 raft_consensus.cc:482] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.879456 20137 raft_consensus.cc:504] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.879912 20137 leader_election.cc:288] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:48.880235 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab213d51c1440819360a63e12aad705" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:48.880263 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab213d51c1440819360a63e12aad705" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:48.880362 15221 raft_consensus.cc:2454] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:48.880411 14958 raft_consensus.cc:2454] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:48.880641 15286 leader_election.cc:302] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:48.880862 20137 raft_consensus.cc:2781] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.880919 20137 raft_consensus.cc:482] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.880967 20137 raft_consensus.cc:3037] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.881889 20137 raft_consensus.cc:504] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.882193 20137 leader_election.cc:288] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:48.882539 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab213d51c1440819360a63e12aad705" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:48.882635 15221 raft_consensus.cc:3037] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.882680 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab213d51c1440819360a63e12aad705" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:48.882836 14958 raft_consensus.cc:3037] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.883613 15221 raft_consensus.cc:2454] T 8ab213d51c1440819360a63e12aad705 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:48.883616 14958 raft_consensus.cc:2454] T 8ab213d51c1440819360a63e12aad705 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:48.883878 15286 leader_election.cc:302] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:48.884037 20137 raft_consensus.cc:2781] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.884256 20137 raft_consensus.cc:686] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:48.884416 20137 consensus_queue.cc:237] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:48.886431 12439 catalog_manager.cc:5484] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:48.927204 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:48.931219 12439 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:48.931367 12439 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:48.933264 12439 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet b88e8ea21db9422b9b54bdc0ad609fbe
I20240501 14:09:48.933779 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet b88e8ea21db9422b9b54bdc0ad609fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:48.933862 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet b88e8ea21db9422b9b54bdc0ad609fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:48.933951 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet b88e8ea21db9422b9b54bdc0ad609fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:48 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:48.934245 20278 tablet_replica.cc:330] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:48.934268 20280 tablet_replica.cc:330] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:48.934403 20278 raft_consensus.cc:2229] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.934473 20279 tablet_replica.cc:330] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:48.934473 20280 raft_consensus.cc:2229] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:48.934587 20278 raft_consensus.cc:2258] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.934610 20279 raft_consensus.cc:2229] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:48.934780 20280 raft_consensus.cc:2258] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.934780 20279 raft_consensus.cc:2258] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:48.935346 20278 ts_tablet_manager.cc:1875] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.935425 20280 ts_tablet_manager.cc:1875] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.935463 20279 ts_tablet_manager.cc:1875] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:48.937870 20280 ts_tablet_manager.cc:1888] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.937988 20280 log.cc:1196] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/b88e8ea21db9422b9b54bdc0ad609fbe
I20240501 14:09:48.938274 20279 ts_tablet_manager.cc:1888] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.938303 20278 ts_tablet_manager.cc:1888] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:48.938354 20280 ts_tablet_manager.cc:1909] T b88e8ea21db9422b9b54bdc0ad609fbe P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:48.938361 20279 log.cc:1196] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/b88e8ea21db9422b9b54bdc0ad609fbe
I20240501 14:09:48.938361 20278 log.cc:1196] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/b88e8ea21db9422b9b54bdc0ad609fbe
I20240501 14:09:48.938717 20278 ts_tablet_manager.cc:1909] T b88e8ea21db9422b9b54bdc0ad609fbe P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:48.938742 20279 ts_tablet_manager.cc:1909] T b88e8ea21db9422b9b54bdc0ad609fbe P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:48.939189 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table_new_name [id=40edc73f1c0e4417a63ffa5dd000e59e]) successfully deleted
I20240501 14:09:48.939400 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table_new_name [id=40edc73f1c0e4417a63ffa5dd000e59e]) successfully deleted
I20240501 14:09:48.939625 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet b88e8ea21db9422b9b54bdc0ad609fbe (table rebalancer_dynamic_table_new_name [id=40edc73f1c0e4417a63ffa5dd000e59e]) successfully deleted
I20240501 14:09:48.956837 20152 raft_consensus.cc:482] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:48.956961 20152 raft_consensus.cc:504] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.957415 20152 leader_election.cc:288] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:48.957778 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca1be8b60de430f8231f9868edfab6e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:48.957801 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca1be8b60de430f8231f9868edfab6e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:48.957923 12979 raft_consensus.cc:2454] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:48.957962 12584 raft_consensus.cc:2454] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:48.958212 15156 leader_election.cc:302] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:48.958410 20152 raft_consensus.cc:2781] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:48.958472 20152 raft_consensus.cc:482] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:48.958513 20152 raft_consensus.cc:3037] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.959740 20152 raft_consensus.cc:504] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.960110 20152 leader_election.cc:288] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:48.960381 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca1be8b60de430f8231f9868edfab6e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:48.960466 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca1be8b60de430f8231f9868edfab6e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:48.960536 12584 raft_consensus.cc:3037] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.960561 12979 raft_consensus.cc:3037] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:48.961575 12979 raft_consensus.cc:2454] T 1ca1be8b60de430f8231f9868edfab6e P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:48.961598 12584 raft_consensus.cc:2454] T 1ca1be8b60de430f8231f9868edfab6e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:48.961944 15156 leader_election.cc:302] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:48.962119 20152 raft_consensus.cc:2781] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:48.962213 20152 raft_consensus.cc:686] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:48.962348 20152 consensus_queue.cc:237] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:48.964866 12443 catalog_manager.cc:5484] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:49.035156 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:49.042059 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 29d62be732cd4075a2dc14ff534c20a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9]), partition=
I20240501 14:09:49.042232 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 29d62be732cd4075a2dc14ff534c20a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9]), partition=
I20240501 14:09:49.042292 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 29d62be732cd4075a2dc14ff534c20a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9]), partition=
I20240501 14:09:49.042343 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d62be732cd4075a2dc14ff534c20a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.042483 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d62be732cd4075a2dc14ff534c20a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.042505 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d62be732cd4075a2dc14ff534c20a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.044386 20266 tablet_bootstrap.cc:492] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:49.044667 20177 tablet_bootstrap.cc:492] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:49.045320 20266 tablet_bootstrap.cc:654] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.045544 20177 tablet_bootstrap.cc:654] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.046648 20292 tablet_bootstrap.cc:492] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:49.046869 20177 tablet_bootstrap.cc:492] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:49.046962 20177 ts_tablet_manager.cc:1367] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.047423 20177 raft_consensus.cc:348] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.047541 20177 raft_consensus.cc:374] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.047590 20177 raft_consensus.cc:729] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.047762 20292 tablet_bootstrap.cc:654] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.047758 20177 consensus_queue.cc:260] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.048113 20177 ts_tablet_manager.cc:1398] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.049587 20292 tablet_bootstrap.cc:492] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:49.049597 20266 tablet_bootstrap.cc:492] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:49.049682 20266 ts_tablet_manager.cc:1367] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:49.049698 20292 ts_tablet_manager.cc:1367] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:49.050108 20292 raft_consensus.cc:348] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.050137 20266 raft_consensus.cc:348] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.050209 20292 raft_consensus.cc:374] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.050231 20266 raft_consensus.cc:374] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.050253 20292 raft_consensus.cc:729] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.050261 20266 raft_consensus.cc:729] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.050369 20266 consensus_queue.cc:260] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.050377 20292 consensus_queue.cc:260] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.050622 20266 ts_tablet_manager.cc:1398] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.050658 20292 ts_tablet_manager.cc:1398] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:49.073647 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.101079 20203 raft_consensus.cc:482] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.101191 20203 raft_consensus.cc:504] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.101650 20203 leader_election.cc:288] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:49.101981 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d62be732cd4075a2dc14ff534c20a1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:49.101981 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d62be732cd4075a2dc14ff534c20a1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:49.102134 14958 raft_consensus.cc:2454] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:49.102134 12849 raft_consensus.cc:2454] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:49.102425 12916 leader_election.cc:302] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:49.102651 20203 raft_consensus.cc:2781] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.102721 20203 raft_consensus.cc:482] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.102797 20203 raft_consensus.cc:3037] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.105445 20203 raft_consensus.cc:504] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.105887 20203 leader_election.cc:288] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:49.106159 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d62be732cd4075a2dc14ff534c20a1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:49.106207 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d62be732cd4075a2dc14ff534c20a1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:49.106266 12849 raft_consensus.cc:3037] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.106329 14958 raft_consensus.cc:3037] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.107086 12849 raft_consensus.cc:2454] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:49.107199 14958 raft_consensus.cc:2454] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:49.107396 12916 leader_election.cc:302] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:49.107553 20203 raft_consensus.cc:2781] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.107692 20203 raft_consensus.cc:686] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:49.107839 20203 consensus_queue.cc:237] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.110483 12440 catalog_manager.cc:5484] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:49.216372 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:49.220700 12849 raft_consensus.cc:1261] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.220808 14958 raft_consensus.cc:1261] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.221086 20203 consensus_queue.cc:1035] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.221213 19736 consensus_queue.cc:1035] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.223579 20247 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.223901 20247 tablet.cc:1620] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:49.224143 20246 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.224397 20246 tablet.cc:1620] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:49.225508 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 29d62be732cd4075a2dc14ff534c20a1 (table rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9]). Expected version 1 got 0
I20240501 14:09:49.226596 12439 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 29d62be732cd4075a2dc14ff534c20a1 (table rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9]). Expected version 1 got 0
I20240501 14:09:49.228227 20133 mvcc.cc:205] Tried to move back new op lower bound from 7022889325444087808 to 7022889324987125760. Current Snapshot: MvccSnapshot[applied={T|T < 7022889325444087808}]
I20240501 14:09:49.231149 12441 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 29d62be732cd4075a2dc14ff534c20a1 (table rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9]). Expected version 1 got 0
I20240501 14:09:49.231451 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9] alter complete (version 1)
I20240501 14:09:49.232805 20309 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.233127 20309 tablet.cc:1620] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
W20240501 14:09:49.234359 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.252189 19736 consensus_queue.cc:1035] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.255899 19736 consensus_queue.cc:1035] T 650c3f2054da4995a3a1668ecae11836 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:49.261713 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.275342 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:49.335943 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:49.341202 20247 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.341572 20247 tablet.cc:1620] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:49.341634 20309 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.341748 20246 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.341944 20309 tablet.cc:1620] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:49.342047 20246 tablet.cc:1620] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:49.348104 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=69522c7f7ab64435815e2871457179f9] alter complete (version 2)
I20240501 14:09:49.379778 20137 consensus_queue.cc:1035] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.384855 20137 consensus_queue.cc:1035] T 8ab213d51c1440819360a63e12aad705 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.424100 19823 consensus_queue.cc:1035] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.427659 19823 consensus_queue.cc:1035] T 1ca1be8b60de430f8231f9868edfab6e P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:49.442878 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.449580 12441 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:49.455147 20247 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:49.455442 20246 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:49.457990 20309 tablet.cc:1722] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:49.462069 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=69522c7f7ab64435815e2871457179f9] alter complete (version 3)
I20240501 14:09:49.494789 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:49.500893 12827 tablet_service.cc:1467] Processing CreateTablet for tablet dfb8bdcad16a48f7b2d459ce6b488238 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.500893 15332 tablet_service.cc:1467] Processing CreateTablet for tablet dfb8bdcad16a48f7b2d459ce6b488238 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.501214 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb8bdcad16a48f7b2d459ce6b488238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.501214 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb8bdcad16a48f7b2d459ce6b488238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.501847 14937 tablet_service.cc:1467] Processing CreateTablet for tablet dfb8bdcad16a48f7b2d459ce6b488238 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.502118 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb8bdcad16a48f7b2d459ce6b488238. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.503520 20292 tablet_bootstrap.cc:492] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:49.504381 20266 tablet_bootstrap.cc:492] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:49.504352 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 0a74abe5f7024268a9f06ae6ac2e67f9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.504343 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 0a74abe5f7024268a9f06ae6ac2e67f9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.504609 20292 tablet_bootstrap.cc:654] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.504616 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a74abe5f7024268a9f06ae6ac2e67f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.504631 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a74abe5f7024268a9f06ae6ac2e67f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.505565 20266 tablet_bootstrap.cc:654] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.505666 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 0a74abe5f7024268a9f06ae6ac2e67f9 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.505940 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a74abe5f7024268a9f06ae6ac2e67f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.506695 20177 tablet_bootstrap.cc:492] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:49.506937 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 3449fc673aea4457a76c4d690df6ae3b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.507200 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3449fc673aea4457a76c4d690df6ae3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.507404 20292 tablet_bootstrap.cc:492] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:49.507493 20292 ts_tablet_manager.cc:1367] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:49.507638 20177 tablet_bootstrap.cc:654] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.507982 20292 raft_consensus.cc:348] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.508085 20292 raft_consensus.cc:374] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.508114 20292 raft_consensus.cc:729] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.508229 20292 consensus_queue.cc:260] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.508392 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 3449fc673aea4457a76c4d690df6ae3b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.508527 20292 ts_tablet_manager.cc:1398] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.508643 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3449fc673aea4457a76c4d690df6ae3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.509557 20336 tablet_bootstrap.cc:492] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:49.510468 20336 tablet_bootstrap.cc:654] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.510658 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 3449fc673aea4457a76c4d690df6ae3b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=73c25cf5d8314854abdddfc0aaeb9e35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:49.510938 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3449fc673aea4457a76c4d690df6ae3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.511674 20266 tablet_bootstrap.cc:492] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:49.511724 20332 tablet_bootstrap.cc:492] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:49.511778 20266 ts_tablet_manager.cc:1367] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:49.512326 20266 raft_consensus.cc:348] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.512461 20266 raft_consensus.cc:374] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.512589 20266 raft_consensus.cc:729] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.512748 20266 consensus_queue.cc:260] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.513090 20266 ts_tablet_manager.cc:1398] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:49.513648 20332 tablet_bootstrap.cc:654] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.515818 20177 tablet_bootstrap.cc:492] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:49.515945 20177 ts_tablet_manager.cc:1367] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 14:09:49.516455 20177 raft_consensus.cc:348] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.516569 20177 raft_consensus.cc:374] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.516603 20177 raft_consensus.cc:729] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.516649 20337 tablet_bootstrap.cc:492] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:49.516731 20177 consensus_queue.cc:260] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.516990 20177 ts_tablet_manager.cc:1398] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:49.518571 20337 tablet_bootstrap.cc:654] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.518934 20336 tablet_bootstrap.cc:492] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:49.519105 20336 ts_tablet_manager.cc:1367] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:49.519665 20336 raft_consensus.cc:348] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.519778 20336 raft_consensus.cc:374] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.519784 20337 tablet_bootstrap.cc:492] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:49.519812 20336 raft_consensus.cc:729] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.519861 20337 ts_tablet_manager.cc:1367] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:49.519984 20336 consensus_queue.cc:260] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.520259 20336 ts_tablet_manager.cc:1398] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:49.520366 20337 raft_consensus.cc:348] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.520483 20337 raft_consensus.cc:374] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.520515 20337 raft_consensus.cc:729] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.520629 20337 consensus_queue.cc:260] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.521530 20337 ts_tablet_manager.cc:1398] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.523172 20332 tablet_bootstrap.cc:492] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:49.523269 20332 ts_tablet_manager.cc:1367] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:49.523636 20336 tablet_bootstrap.cc:492] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:49.523748 20332 raft_consensus.cc:348] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.523885 20332 raft_consensus.cc:374] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.523968 20332 raft_consensus.cc:729] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.524103 20332 consensus_queue.cc:260] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.524382 20332 ts_tablet_manager.cc:1398] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.524572 20332 tablet_bootstrap.cc:492] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:49.524734 20336 tablet_bootstrap.cc:654] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.525686 20332 tablet_bootstrap.cc:654] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.525806 20336 tablet_bootstrap.cc:492] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:49.525884 20336 ts_tablet_manager.cc:1367] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:49.526343 20336 raft_consensus.cc:348] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.526459 20336 raft_consensus.cc:374] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.526491 20336 raft_consensus.cc:729] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.526643 20336 consensus_queue.cc:260] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.526993 20336 ts_tablet_manager.cc:1398] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.528985 20332 tablet_bootstrap.cc:492] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:49.529135 20332 ts_tablet_manager.cc:1367] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:49.529603 20332 raft_consensus.cc:348] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.529721 20332 raft_consensus.cc:374] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.529752 20332 raft_consensus.cc:729] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.529888 20332 consensus_queue.cc:260] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.530194 20332 ts_tablet_manager.cc:1398] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.530570 20332 tablet_bootstrap.cc:492] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:49.531540 20332 tablet_bootstrap.cc:654] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.532626 20332 tablet_bootstrap.cc:492] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:49.532704 20332 ts_tablet_manager.cc:1367] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:49.533146 20332 raft_consensus.cc:348] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.533250 20332 raft_consensus.cc:374] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.533282 20332 raft_consensus.cc:729] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.533454 20332 consensus_queue.cc:260] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.533825 20332 ts_tablet_manager.cc:1398] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:49.555209 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.564846 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:49.565002 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:49.567108 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 29d62be732cd4075a2dc14ff534c20a1
I20240501 14:09:49.567780 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d62be732cd4075a2dc14ff534c20a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:49 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:49.567782 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d62be732cd4075a2dc14ff534c20a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:49 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:49.567813 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d62be732cd4075a2dc14ff534c20a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:49 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:49.568142 20345 tablet_replica.cc:330] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:49.568174 20346 tablet_replica.cc:330] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:49.568323 20345 raft_consensus.cc:2229] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.568356 20346 raft_consensus.cc:2229] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:49.568446 20345 raft_consensus.cc:2258] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.568488 20346 raft_consensus.cc:2258] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.568699 20344 tablet_replica.cc:330] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:49.568864 20344 raft_consensus.cc:2229] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:49.569077 20344 raft_consensus.cc:2258] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:49.569104 20346 ts_tablet_manager.cc:1875] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:49.569104 20345 ts_tablet_manager.cc:1875] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:49.569579 20344 ts_tablet_manager.cc:1875] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:49.571655 20344 ts_tablet_manager.cc:1888] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:49.571748 20344 log.cc:1196] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/29d62be732cd4075a2dc14ff534c20a1
I20240501 14:09:49.571868 20346 ts_tablet_manager.cc:1888] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:49.571954 20346 log.cc:1196] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/29d62be732cd4075a2dc14ff534c20a1
I20240501 14:09:49.572315 20346 ts_tablet_manager.cc:1909] T 29d62be732cd4075a2dc14ff534c20a1 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:49.572388 20344 ts_tablet_manager.cc:1909] T 29d62be732cd4075a2dc14ff534c20a1 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:49.572676 20345 ts_tablet_manager.cc:1888] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:49.572759 20345 log.cc:1196] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/29d62be732cd4075a2dc14ff534c20a1
I20240501 14:09:49.573007 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 29d62be732cd4075a2dc14ff534c20a1 (table rebalancer_dynamic_table_new_name [id=69522c7f7ab64435815e2871457179f9]) successfully deleted
I20240501 14:09:49.573163 20345 ts_tablet_manager.cc:1909] T 29d62be732cd4075a2dc14ff534c20a1 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:49.573274 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 29d62be732cd4075a2dc14ff534c20a1 (table rebalancer_dynamic_table_new_name [id=69522c7f7ab64435815e2871457179f9]) successfully deleted
I20240501 14:09:49.573777 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 29d62be732cd4075a2dc14ff534c20a1 (table rebalancer_dynamic_table_new_name [id=69522c7f7ab64435815e2871457179f9]) successfully deleted
W20240501 14:09:49.610275 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.635510 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.642843 19584 raft_consensus.cc:482] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.642941 19584 raft_consensus.cc:504] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.643368 19584 leader_election.cc:288] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:49.643724 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a74abe5f7024268a9f06ae6ac2e67f9" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:49.643738 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a74abe5f7024268a9f06ae6ac2e67f9" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:49.643852 12978 raft_consensus.cc:2454] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:49.643857 15351 raft_consensus.cc:2454] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:49.644137 12654 leader_election.cc:302] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:49.644302 19584 raft_consensus.cc:2781] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.644379 19584 raft_consensus.cc:482] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.644418 19584 raft_consensus.cc:3037] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.645237 19584 raft_consensus.cc:504] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.645551 19584 leader_election.cc:288] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:49.645946 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a74abe5f7024268a9f06ae6ac2e67f9" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:49.646076 12978 raft_consensus.cc:3037] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.646188 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a74abe5f7024268a9f06ae6ac2e67f9" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:49.646293 15351 raft_consensus.cc:3037] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.647193 12978 raft_consensus.cc:2454] T 0a74abe5f7024268a9f06ae6ac2e67f9 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
W20240501 14:09:49.647209 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.647244 15351 raft_consensus.cc:2454] T 0a74abe5f7024268a9f06ae6ac2e67f9 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:49.647496 12653 leader_election.cc:302] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:49.647675 19584 raft_consensus.cc:2781] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.647768 19584 raft_consensus.cc:686] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
W20240501 14:09:49.647920 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:49.647905 19584 consensus_queue.cc:237] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.649956 12443 catalog_manager.cc:5484] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:49.668915 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:49.674178 12698 tablet_service.cc:1467] Processing CreateTablet for tablet cc9d421c4e7348d19d8b8ab913066bc1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]), partition=
I20240501 14:09:49.674193 12958 tablet_service.cc:1467] Processing CreateTablet for tablet cc9d421c4e7348d19d8b8ab913066bc1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]), partition=
I20240501 14:09:49.674448 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9d421c4e7348d19d8b8ab913066bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.674453 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9d421c4e7348d19d8b8ab913066bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.675256 12827 tablet_service.cc:1467] Processing CreateTablet for tablet cc9d421c4e7348d19d8b8ab913066bc1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]), partition=
I20240501 14:09:49.675496 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9d421c4e7348d19d8b8ab913066bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:49.676620 20336 tablet_bootstrap.cc:492] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:49.676635 20177 tablet_bootstrap.cc:492] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:49.677296 20177 tablet_bootstrap.cc:654] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.677495 20292 tablet_bootstrap.cc:492] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:49.677503 20336 tablet_bootstrap.cc:654] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.678386 20292 tablet_bootstrap.cc:654] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:49.679076 20336 tablet_bootstrap.cc:492] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:49.679167 20336 ts_tablet_manager.cc:1367] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:49.679482 20292 tablet_bootstrap.cc:492] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:49.679553 20292 ts_tablet_manager.cc:1367] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:49.679538 20336 raft_consensus.cc:348] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.679615 20336 raft_consensus.cc:374] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.679636 20336 raft_consensus.cc:729] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.679735 20336 consensus_queue.cc:260] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.679955 20336 ts_tablet_manager.cc:1398] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:49.679939 20292 raft_consensus.cc:348] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.680047 20292 raft_consensus.cc:374] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.680089 20292 raft_consensus.cc:729] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.680207 20292 consensus_queue.cc:260] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.680570 20292 ts_tablet_manager.cc:1398] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:49.682307 20177 tablet_bootstrap.cc:492] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:49.682418 20177 ts_tablet_manager.cc:1367] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:09:49.682439 20137 raft_consensus.cc:482] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.682518 20137 raft_consensus.cc:504] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.682866 20137 leader_election.cc:288] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:49.682931 20177 raft_consensus.cc:348] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.683059 20177 raft_consensus.cc:374] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:49.683090 20177 raft_consensus.cc:729] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:49.683209 20177 consensus_queue.cc:260] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.683319 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb8bdcad16a48f7b2d459ce6b488238" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:49.683449 14958 raft_consensus.cc:2454] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:49.683495 20177 ts_tablet_manager.cc:1398] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:49.683751 15287 leader_election.cc:302] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:49.683926 20137 raft_consensus.cc:2781] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.683980 20137 raft_consensus.cc:482] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.684036 20137 raft_consensus.cc:3037] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.684861 20137 raft_consensus.cc:504] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.685182 20137 leader_election.cc:288] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:49.685595 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb8bdcad16a48f7b2d459ce6b488238" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:49.685710 12849 raft_consensus.cc:2454] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:49.685894 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb8bdcad16a48f7b2d459ce6b488238" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:49.685982 12848 raft_consensus.cc:3037] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.686791 12848 raft_consensus.cc:2454] T dfb8bdcad16a48f7b2d459ce6b488238 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:49.686890 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb8bdcad16a48f7b2d459ce6b488238" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:49.687008 14958 raft_consensus.cc:3037] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.687884 14958 raft_consensus.cc:2454] T dfb8bdcad16a48f7b2d459ce6b488238 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:49.688141 15287 leader_election.cc:302] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:49.688299 20137 raft_consensus.cc:2781] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.688378 20137 raft_consensus.cc:686] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:49.688525 20137 consensus_queue.cc:237] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:49.691411 12443 catalog_manager.cc:5484] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:49.764859 20203 raft_consensus.cc:482] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.764976 20203 raft_consensus.cc:504] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.765343 20203 leader_election.cc:288] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:49.765682 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9d421c4e7348d19d8b8ab913066bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:49.765739 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9d421c4e7348d19d8b8ab913066bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:49.765825 12717 raft_consensus.cc:2454] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:49.765901 12848 raft_consensus.cc:2454] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:49.766153 12913 leader_election.cc:302] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:49.766796 20203 raft_consensus.cc:2781] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.766865 20203 raft_consensus.cc:482] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.766906 20203 raft_consensus.cc:3037] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.767966 20203 raft_consensus.cc:504] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.768280 20203 leader_election.cc:288] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:49.768589 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9d421c4e7348d19d8b8ab913066bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:49.768602 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9d421c4e7348d19d8b8ab913066bc1" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:49.768703 12717 raft_consensus.cc:3037] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.768726 12848 raft_consensus.cc:3037] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.769495 12717 raft_consensus.cc:2454] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:49.769570 12848 raft_consensus.cc:2454] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:49.769789 12913 leader_election.cc:302] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:49.770009 20203 raft_consensus.cc:2781] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.770128 20203 raft_consensus.cc:686] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:49.770263 20203 consensus_queue.cc:237] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:49.772316 12443 catalog_manager.cc:5484] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:49.780715 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.807952 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:49.828694 20226 raft_consensus.cc:482] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:49.828804 20226 raft_consensus.cc:504] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.829136 20226 leader_election.cc:288] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:49.829499 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3449fc673aea4457a76c4d690df6ae3b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:49.829671 12585 raft_consensus.cc:2454] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:49.829620 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3449fc673aea4457a76c4d690df6ae3b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:49.829782 15351 raft_consensus.cc:2454] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:49.829959 12653 leader_election.cc:302] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:49.830144 20226 raft_consensus.cc:2781] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:49.830200 20226 raft_consensus.cc:482] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:49.830226 20226 raft_consensus.cc:3037] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.831182 20226 raft_consensus.cc:504] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.831467 20226 leader_election.cc:288] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:49.831787 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3449fc673aea4457a76c4d690df6ae3b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:49.831808 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3449fc673aea4457a76c4d690df6ae3b" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:49.831905 15351 raft_consensus.cc:3037] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.831907 12585 raft_consensus.cc:3037] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:49.832870 12585 raft_consensus.cc:2454] T 3449fc673aea4457a76c4d690df6ae3b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:49.832870 15351 raft_consensus.cc:2454] T 3449fc673aea4457a76c4d690df6ae3b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:49.833294 12653 leader_election.cc:302] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:49.833529 20226 raft_consensus.cc:2781] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:49.833621 20226 raft_consensus.cc:686] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:49.833745 20226 consensus_queue.cc:237] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:49.836017 12443 catalog_manager.cc:5484] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:49.850394 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.855713 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.868021 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:09:49.877038 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:49.881028 12717 raft_consensus.cc:1261] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.881067 12848 raft_consensus.cc:1261] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:49.881537 19736 consensus_queue.cc:1035] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.881687 20203 consensus_queue.cc:1035] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:49.884688 12441 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]). Expected version 1 got 0
I20240501 14:09:49.885226 20247 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.885581 20247 tablet.cc:1620] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:49.886333 20246 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.886649 20246 tablet.cc:1620] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:49.886703 12441 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]). Expected version 1 got 0
I20240501 14:09:49.888729 12443 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]). Expected version 1 got 0
I20240501 14:09:49.889127 12440 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6]). Expected version 1 got 0
I20240501 14:09:49.890791 20376 mvcc.cc:205] Tried to move back new op lower bound from 7022889328148815872 to 7022889327700410368. Current Snapshot: MvccSnapshot[applied={T|T < 7022889328148815872}]
I20240501 14:09:49.891074 20379 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:49.891283 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6] alter complete (version 1)
I20240501 14:09:49.891778 20379 tablet.cc:1620] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
W20240501 14:09:49.911845 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.940552 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:49.987617 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:49.992982 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:49.998032 20247 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.998380 20247 tablet.cc:1620] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:49.998453 20379 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.999106 20379 tablet.cc:1620] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:49.999174 20246 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:49.999445 20246 tablet.cc:1620] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
W20240501 14:09:50.002990 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.005131 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=01eefaf4d86343d1a841d830bee3e8d6] alter complete (version 2)
I20240501 14:09:50.025593 20226 consensus_queue.cc:1035] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.030381 20381 consensus_queue.cc:1035] T 0a74abe5f7024268a9f06ae6ac2e67f9 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.106491 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:50.111416 20247 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.111953 20379 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.112818 20246 tablet.cc:1722] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.117528 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=01eefaf4d86343d1a841d830bee3e8d6] alter complete (version 3)
W20240501 14:09:50.122457 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.176638 20276 consensus_queue.cc:1035] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.180675 20137 consensus_queue.cc:1035] T dfb8bdcad16a48f7b2d459ce6b488238 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:50.208581 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.220173 12441 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.220338 12441 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.222782 12441 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet cc9d421c4e7348d19d8b8ab913066bc1
I20240501 14:09:50.223764 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9d421c4e7348d19d8b8ab913066bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:50.223939 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9d421c4e7348d19d8b8ab913066bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:50.224023 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9d421c4e7348d19d8b8ab913066bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:50.224120 20397 tablet_replica.cc:330] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:50.224264 20397 raft_consensus.cc:2229] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:50.224269 20398 tablet_replica.cc:330] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:50.224357 20399 tablet_replica.cc:330] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:50.224416 20398 raft_consensus.cc:2229] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.224493 20397 raft_consensus.cc:2258] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.224519 20399 raft_consensus.cc:2229] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.224655 20398 raft_consensus.cc:2258] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.224875 20399 raft_consensus.cc:2258] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.225351 20397 ts_tablet_manager.cc:1875] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.225373 20398 ts_tablet_manager.cc:1875] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.225659 20399 ts_tablet_manager.cc:1875] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.227537 20397 ts_tablet_manager.cc:1888] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.227648 20397 log.cc:1196] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/cc9d421c4e7348d19d8b8ab913066bc1
I20240501 14:09:50.227737 20398 ts_tablet_manager.cc:1888] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.227737 20399 ts_tablet_manager.cc:1888] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.227830 20399 log.cc:1196] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc9d421c4e7348d19d8b8ab913066bc1
I20240501 14:09:50.227885 20398 log.cc:1196] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/cc9d421c4e7348d19d8b8ab913066bc1
I20240501 14:09:50.228015 20397 ts_tablet_manager.cc:1909] T cc9d421c4e7348d19d8b8ab913066bc1 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:50.228199 20399 ts_tablet_manager.cc:1909] T cc9d421c4e7348d19d8b8ab913066bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:50.228220 20398 ts_tablet_manager.cc:1909] T cc9d421c4e7348d19d8b8ab913066bc1 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:50.228762 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table_new_name [id=01eefaf4d86343d1a841d830bee3e8d6]) successfully deleted
I20240501 14:09:50.228969 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table_new_name [id=01eefaf4d86343d1a841d830bee3e8d6]) successfully deleted
I20240501 14:09:50.229308 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet cc9d421c4e7348d19d8b8ab913066bc1 (table rebalancer_dynamic_table_new_name [id=01eefaf4d86343d1a841d830bee3e8d6]) successfully deleted
W20240501 14:09:50.247152 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:50.324915 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:50.329882 12827 tablet_service.cc:1467] Processing CreateTablet for tablet f6aad98d5f3d49e494ace70451e70142 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42]), partition=
I20240501 14:09:50.329934 14937 tablet_service.cc:1467] Processing CreateTablet for tablet f6aad98d5f3d49e494ace70451e70142 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42]), partition=
I20240501 14:09:50.330139 12958 tablet_service.cc:1467] Processing CreateTablet for tablet f6aad98d5f3d49e494ace70451e70142 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42]), partition=
I20240501 14:09:50.330197 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aad98d5f3d49e494ace70451e70142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.330199 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aad98d5f3d49e494ace70451e70142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.330348 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6aad98d5f3d49e494ace70451e70142. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.332624 20412 tablet_bootstrap.cc:492] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:50.332780 20410 tablet_bootstrap.cc:492] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:50.333045 20411 tablet_bootstrap.cc:492] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:50.333606 20410 tablet_bootstrap.cc:654] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.333614 20412 tablet_bootstrap.cc:654] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.334048 20411 tablet_bootstrap.cc:654] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.334854 20410 tablet_bootstrap.cc:492] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:50.334990 20410 ts_tablet_manager.cc:1367] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:50.335096 20412 tablet_bootstrap.cc:492] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:50.335160 20411 tablet_bootstrap.cc:492] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:50.335235 20412 ts_tablet_manager.cc:1367] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:50.335247 20411 ts_tablet_manager.cc:1367] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:50.335747 20411 raft_consensus.cc:348] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.335866 20411 raft_consensus.cc:374] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.335912 20411 raft_consensus.cc:729] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.336069 20411 consensus_queue.cc:260] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.336369 20411 ts_tablet_manager.cc:1398] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.336719 20410 raft_consensus.cc:348] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.336828 20410 raft_consensus.cc:374] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.336920 20410 raft_consensus.cc:729] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.336234 20412 raft_consensus.cc:348] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.337311 20412 raft_consensus.cc:374] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.337396 20412 raft_consensus.cc:729] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.337103 20410 consensus_queue.cc:260] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.337558 20412 consensus_queue.cc:260] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.337857 20412 ts_tablet_manager.cc:1398] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:50.339820 20410 ts_tablet_manager.cc:1398] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:50.373497 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:50.380343 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 628137adc39c4ce29342326a171c30d7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.380326 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 628137adc39c4ce29342326a171c30d7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.380435 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 628137adc39c4ce29342326a171c30d7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.380659 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628137adc39c4ce29342326a171c30d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.380657 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628137adc39c4ce29342326a171c30d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.380671 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628137adc39c4ce29342326a171c30d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.382908 20412 tablet_bootstrap.cc:492] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:50.383122 12566 tablet_service.cc:1467] Processing CreateTablet for tablet fd2686b167a941a4ae94881663197015 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.383198 14937 tablet_service.cc:1467] Processing CreateTablet for tablet fd2686b167a941a4ae94881663197015 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.383436 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2686b167a941a4ae94881663197015. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.383442 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2686b167a941a4ae94881663197015. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.383967 20412 tablet_bootstrap.cc:654] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.386215 20411 tablet_bootstrap.cc:492] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:50.387512 15201 tablet_service.cc:1467] Processing CreateTablet for tablet ba7850968c07445ca18f9a9729e0a134 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.387804 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7850968c07445ca18f9a9729e0a134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.387745 12696 tablet_service.cc:1467] Processing CreateTablet for tablet ba7850968c07445ca18f9a9729e0a134 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.388041 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7850968c07445ca18f9a9729e0a134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.387512 15202 tablet_service.cc:1467] Processing CreateTablet for tablet fd2686b167a941a4ae94881663197015 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.388296 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2686b167a941a4ae94881663197015. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.388784 20418 tablet_bootstrap.cc:492] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:50.389819 20418 tablet_bootstrap.cc:654] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.391798 20411 tablet_bootstrap.cc:654] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.391935 20419 tablet_bootstrap.cc:492] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:50.393393 20419 tablet_bootstrap.cc:654] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.393463 20416 tablet_bootstrap.cc:492] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:50.393661 20411 tablet_bootstrap.cc:492] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:50.393754 20411 ts_tablet_manager.cc:1367] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:50.394225 20411 raft_consensus.cc:348] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.394335 20411 raft_consensus.cc:374] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.394363 20411 raft_consensus.cc:729] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.394484 20411 consensus_queue.cc:260] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.394807 20411 ts_tablet_manager.cc:1398] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.394891 20416 tablet_bootstrap.cc:654] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.394968 20419 tablet_bootstrap.cc:492] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:50.395045 20419 ts_tablet_manager.cc:1367] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:50.395340 12827 tablet_service.cc:1467] Processing CreateTablet for tablet ba7850968c07445ca18f9a9729e0a134 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=2d08201978d3411e8884d1df2d304ed9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:50.395591 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7850968c07445ca18f9a9729e0a134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:50.395538 20419 raft_consensus.cc:348] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.395658 20419 raft_consensus.cc:374] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.395696 20419 raft_consensus.cc:729] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.395836 20419 consensus_queue.cc:260] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.396081 20416 tablet_bootstrap.cc:492] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:50.396095 20419 ts_tablet_manager.cc:1398] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.396163 20416 ts_tablet_manager.cc:1367] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:50.396260 20419 tablet_bootstrap.cc:492] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:50.396590 20416 raft_consensus.cc:348] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.396698 20416 raft_consensus.cc:374] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.396730 20416 raft_consensus.cc:729] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.396838 20416 consensus_queue.cc:260] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.397130 20419 tablet_bootstrap.cc:654] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.397150 20416 ts_tablet_manager.cc:1398] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:50.403018 20419 tablet_bootstrap.cc:492] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:50.403121 20419 ts_tablet_manager.cc:1367] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:50.403579 20419 raft_consensus.cc:348] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.403685 20419 raft_consensus.cc:374] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.403738 20419 raft_consensus.cc:729] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.403879 20419 consensus_queue.cc:260] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.404191 20419 ts_tablet_manager.cc:1398] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.407011 20420 tablet_bootstrap.cc:492] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:50.408084 19584 consensus_queue.cc:1035] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.408902 20420 tablet_bootstrap.cc:654] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.410363 20412 tablet_bootstrap.cc:492] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:50.410480 20412 ts_tablet_manager.cc:1367] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240501 14:09:50.410941 20412 raft_consensus.cc:348] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.411051 20412 raft_consensus.cc:374] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.411079 20412 raft_consensus.cc:729] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.411206 20412 consensus_queue.cc:260] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.411495 20412 ts_tablet_manager.cc:1398] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.411649 20412 tablet_bootstrap.cc:492] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:50.412092 20420 tablet_bootstrap.cc:492] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:50.412230 20420 ts_tablet_manager.cc:1367] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:50.412557 20412 tablet_bootstrap.cc:654] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.412719 20420 raft_consensus.cc:348] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.412840 20420 raft_consensus.cc:374] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.412875 20420 raft_consensus.cc:729] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.413028 20420 consensus_queue.cc:260] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.413341 20420 ts_tablet_manager.cc:1398] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.413486 20420 tablet_bootstrap.cc:492] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:50.414323 20420 tablet_bootstrap.cc:654] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:50.415889 19584 consensus_queue.cc:1035] T 3449fc673aea4457a76c4d690df6ae3b P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.418991 20418 tablet_bootstrap.cc:492] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:50.419092 20418 ts_tablet_manager.cc:1367] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.001s
I20240501 14:09:50.419539 20418 raft_consensus.cc:348] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.419660 20418 raft_consensus.cc:374] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.419687 20418 raft_consensus.cc:729] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.419796 20418 consensus_queue.cc:260] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.420035 20418 ts_tablet_manager.cc:1398] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:50.421198 20412 tablet_bootstrap.cc:492] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:50.421286 20412 ts_tablet_manager.cc:1367] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:50.421782 20412 raft_consensus.cc:348] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.421887 20412 raft_consensus.cc:374] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.421918 20412 raft_consensus.cc:729] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.422040 20412 consensus_queue.cc:260] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.422283 20412 ts_tablet_manager.cc:1398] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.425776 20420 tablet_bootstrap.cc:492] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:50.425863 20420 ts_tablet_manager.cc:1367] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:50.426260 20420 raft_consensus.cc:348] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.426354 20420 raft_consensus.cc:374] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:50.426376 20420 raft_consensus.cc:729] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:50.426540 20420 consensus_queue.cc:260] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.426898 20420 ts_tablet_manager.cc:1398] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:50.433864 19971 raft_consensus.cc:482] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.433977 19971 raft_consensus.cc:504] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.434338 19971 leader_election.cc:288] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:50.434752 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aad98d5f3d49e494ace70451e70142" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:50.434784 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aad98d5f3d49e494ace70451e70142" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:50.434892 14958 raft_consensus.cc:2454] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:50.434892 12978 raft_consensus.cc:2454] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:50.435204 12784 leader_election.cc:302] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:50.435375 19971 raft_consensus.cc:2781] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.435446 19971 raft_consensus.cc:482] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.435485 19971 raft_consensus.cc:3037] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.436476 19971 raft_consensus.cc:504] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.436787 19971 leader_election.cc:288] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:50.437170 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aad98d5f3d49e494ace70451e70142" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:50.437286 14958 raft_consensus.cc:3037] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.437253 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6aad98d5f3d49e494ace70451e70142" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:50.437364 12978 raft_consensus.cc:3037] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.438318 12978 raft_consensus.cc:2454] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:50.438330 14958 raft_consensus.cc:2454] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:50.438699 12783 leader_election.cc:302] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:50.439379 20361 raft_consensus.cc:2781] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.439599 20361 raft_consensus.cc:686] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:50.439743 20361 consensus_queue.cc:237] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:50.442090 12441 catalog_manager.cc:5484] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:50.444888 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.456363 20152 raft_consensus.cc:482] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.456473 20152 raft_consensus.cc:504] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.456959 20152 leader_election.cc:288] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:50.457304 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7850968c07445ca18f9a9729e0a134" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:50.457376 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7850968c07445ca18f9a9729e0a134" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:50.457476 12849 raft_consensus.cc:2454] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:50.457540 12717 raft_consensus.cc:2454] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:50.457798 15157 leader_election.cc:302] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:50.458016 20152 raft_consensus.cc:2781] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.458076 20152 raft_consensus.cc:482] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.458112 20152 raft_consensus.cc:3037] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.459043 20152 raft_consensus.cc:504] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.459362 20152 leader_election.cc:288] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:50.459653 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7850968c07445ca18f9a9729e0a134" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:50.459707 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7850968c07445ca18f9a9729e0a134" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:50.459757 12849 raft_consensus.cc:3037] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.459803 12717 raft_consensus.cc:3037] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.460572 12849 raft_consensus.cc:2454] T ba7850968c07445ca18f9a9729e0a134 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:50.460712 12717 raft_consensus.cc:2454] T ba7850968c07445ca18f9a9729e0a134 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:50.460846 15157 leader_election.cc:302] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:50.461047 20152 raft_consensus.cc:2781] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.461133 20152 raft_consensus.cc:686] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:50.461266 20152 consensus_queue.cc:237] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:50.463449 12441 catalog_manager.cc:5484] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:50.566833 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:50.570659 14958 raft_consensus.cc:1261] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:50.570778 12978 raft_consensus.cc:1261] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:50.571174 20361 consensus_queue.cc:1035] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.571322 19971 consensus_queue.cc:1035] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.573922 20246 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:50.574242 20246 tablet.cc:1620] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:50.574297 20247 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:50.574611 20247 tablet.cc:1620] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:50.575340 12439 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet f6aad98d5f3d49e494ace70451e70142 (table rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42]). Expected version 1 got 0
I20240501 14:09:50.575860 12441 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet f6aad98d5f3d49e494ace70451e70142 (table rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42]). Expected version 1 got 0
I20240501 14:09:50.579576 20447 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:50.579788 12440 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet f6aad98d5f3d49e494ace70451e70142 (table rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42]). Expected version 1 got 0
I20240501 14:09:50.580310 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42] alter complete (version 1)
I20240501 14:09:50.580299 20447 tablet.cc:1620] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:50.587075 20315 raft_consensus.cc:482] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.587185 20315 raft_consensus.cc:504] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.587671 20315 leader_election.cc:288] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:50.588198 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2686b167a941a4ae94881663197015" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:50.588331 15222 raft_consensus.cc:2454] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:50.588366 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2686b167a941a4ae94881663197015" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:50.588501 12585 raft_consensus.cc:2454] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:50.588634 14891 leader_election.cc:302] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:50.588896 20315 raft_consensus.cc:2781] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.588953 20315 raft_consensus.cc:482] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.589015 20315 raft_consensus.cc:3037] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.589934 20315 raft_consensus.cc:504] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.590312 20315 leader_election.cc:288] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:50.590591 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2686b167a941a4ae94881663197015" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:50.590700 15222 raft_consensus.cc:3037] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.590811 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2686b167a941a4ae94881663197015" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:50.590930 12585 raft_consensus.cc:3037] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.591667 15222 raft_consensus.cc:2454] T fd2686b167a941a4ae94881663197015 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:50.591928 14891 leader_election.cc:302] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:50.591975 12585 raft_consensus.cc:2454] T fd2686b167a941a4ae94881663197015 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:50.592094 20315 raft_consensus.cc:2781] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.592283 20315 raft_consensus.cc:686] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:50.592442 20315 consensus_queue.cc:237] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:50.595090 12439 catalog_manager.cc:5484] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:50.607256 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:50.632184 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.685345 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:50.692440 20246 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.692831 20246 tablet.cc:1620] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:50.693786 20447 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.694106 20447 tablet.cc:1620] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:50.696197 20247 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:50.696530 20247 tablet.cc:1620] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:50.701227 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=8347e4ba3a994d148b82fb63910d5c42] alter complete (version 2)
I20240501 14:09:50.720453 20381 raft_consensus.cc:482] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:50.720567 20381 raft_consensus.cc:504] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.720948 20381 leader_election.cc:288] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:50.721328 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628137adc39c4ce29342326a171c30d7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:50.721498 15352 raft_consensus.cc:2454] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:50.721508 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628137adc39c4ce29342326a171c30d7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:50.721644 12849 raft_consensus.cc:2454] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:50.721866 12654 leader_election.cc:302] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:50.722038 20381 raft_consensus.cc:2781] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:50.722097 20381 raft_consensus.cc:482] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:50.722136 20381 raft_consensus.cc:3037] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.723176 20381 raft_consensus.cc:504] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.723471 20381 leader_election.cc:288] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:50.723838 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628137adc39c4ce29342326a171c30d7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:50.723970 12849 raft_consensus.cc:3037] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.723923 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628137adc39c4ce29342326a171c30d7" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:50.724067 15352 raft_consensus.cc:3037] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:50.724942 12849 raft_consensus.cc:2454] T 628137adc39c4ce29342326a171c30d7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:50.725162 15352 raft_consensus.cc:2454] T 628137adc39c4ce29342326a171c30d7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:50.725324 12654 leader_election.cc:302] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:50.725493 20381 raft_consensus.cc:2781] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:50.725605 20381 raft_consensus.cc:686] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:50.725744 20381 consensus_queue.cc:237] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:50.728127 12439 catalog_manager.cc:5484] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:50.736423 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:50.759303 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.804224 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:50.810518 20246 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.810969 20447 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:50.811954 20247 tablet.cc:1722] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 14:09:50.816659 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:50.816658 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=8347e4ba3a994d148b82fb63910d5c42] alter complete (version 3)
I20240501 14:09:50.918593 12439 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.918794 12439 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:50.920840 12439 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet f6aad98d5f3d49e494ace70451e70142
I20240501 14:09:50.921581 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aad98d5f3d49e494ace70451e70142 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:50.921660 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aad98d5f3d49e494ace70451e70142 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:50.921666 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet f6aad98d5f3d49e494ace70451e70142 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:50 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:50.921948 20477 tablet_replica.cc:330] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:50.922019 20476 tablet_replica.cc:330] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:50.922223 20476 raft_consensus.cc:2229] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:50.922224 20477 raft_consensus.cc:2229] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.922500 20478 tablet_replica.cc:330] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:50.922627 20477 raft_consensus.cc:2258] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.922667 20478 raft_consensus.cc:2229] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:50.922683 20476 raft_consensus.cc:2258] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.922838 20478 raft_consensus.cc:2258] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:50.923354 20477 ts_tablet_manager.cc:1875] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.923354 20478 ts_tablet_manager.cc:1875] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.923974 20476 ts_tablet_manager.cc:1875] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:50.925653 20477 ts_tablet_manager.cc:1888] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.925653 20478 ts_tablet_manager.cc:1888] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.925732 20477 log.cc:1196] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/f6aad98d5f3d49e494ace70451e70142
I20240501 14:09:50.925732 20478 log.cc:1196] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/f6aad98d5f3d49e494ace70451e70142
I20240501 14:09:50.925791 20476 ts_tablet_manager.cc:1888] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:50.925876 20476 log.cc:1196] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/f6aad98d5f3d49e494ace70451e70142
I20240501 14:09:50.926033 20478 ts_tablet_manager.cc:1909] T f6aad98d5f3d49e494ace70451e70142 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:50.926048 20477 ts_tablet_manager.cc:1909] T f6aad98d5f3d49e494ace70451e70142 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:50.926271 20476 ts_tablet_manager.cc:1909] T f6aad98d5f3d49e494ace70451e70142 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:50.926824 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet f6aad98d5f3d49e494ace70451e70142 (table rebalancer_dynamic_table_new_name [id=8347e4ba3a994d148b82fb63910d5c42]) successfully deleted
I20240501 14:09:50.926903 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet f6aad98d5f3d49e494ace70451e70142 (table rebalancer_dynamic_table_new_name [id=8347e4ba3a994d148b82fb63910d5c42]) successfully deleted
I20240501 14:09:50.926923 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet f6aad98d5f3d49e494ace70451e70142 (table rebalancer_dynamic_table_new_name [id=8347e4ba3a994d148b82fb63910d5c42]) successfully deleted
I20240501 14:09:50.936663 20330 consensus_queue.cc:1035] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:50.940176 20152 consensus_queue.cc:1035] T ba7850968c07445ca18f9a9729e0a134 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:50.957811 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:51.023555 12441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:51.029218 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 2a802a43ca4c43ab91f555433ca24ba2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ecef44c13b447fc900641841726f100]), partition=
I20240501 14:09:51.029439 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 2a802a43ca4c43ab91f555433ca24ba2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ecef44c13b447fc900641841726f100]), partition=
I20240501 14:09:51.029517 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a802a43ca4c43ab91f555433ca24ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.029691 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a802a43ca4c43ab91f555433ca24ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.032027 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 2a802a43ca4c43ab91f555433ca24ba2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ecef44c13b447fc900641841726f100]), partition=
I20240501 14:09:51.032086 20484 tablet_bootstrap.cc:492] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:51.032263 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a802a43ca4c43ab91f555433ca24ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.033041 20484 tablet_bootstrap.cc:654] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.034325 20484 tablet_bootstrap.cc:492] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:51.034399 20486 tablet_bootstrap.cc:492] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:51.034407 20484 ts_tablet_manager.cc:1367] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:51.034981 20483 tablet_bootstrap.cc:492] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:51.035483 20484 raft_consensus.cc:348] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.035642 20484 raft_consensus.cc:374] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.035699 20484 raft_consensus.cc:729] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.035984 20484 consensus_queue.cc:260] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.036166 20483 tablet_bootstrap.cc:654] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.036387 20484 ts_tablet_manager.cc:1398] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.036579 20486 tablet_bootstrap.cc:654] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.037889 20483 tablet_bootstrap.cc:492] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:51.037979 20483 ts_tablet_manager.cc:1367] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:51.038378 20483 raft_consensus.cc:348] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.039412 20483 raft_consensus.cc:374] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.039465 20483 raft_consensus.cc:729] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.039594 20483 consensus_queue.cc:260] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.039876 20483 ts_tablet_manager.cc:1398] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.054670 20486 tablet_bootstrap.cc:492] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:51.054812 20486 ts_tablet_manager.cc:1367] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:51.055335 20486 raft_consensus.cc:348] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.055437 20486 raft_consensus.cc:374] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.055464 20486 raft_consensus.cc:729] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.055615 20486 consensus_queue.cc:260] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.055889 20486 ts_tablet_manager.cc:1398] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.103724 20315 raft_consensus.cc:482] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.103853 20315 raft_consensus.cc:504] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.104295 20315 leader_election.cc:288] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:51.104799 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a802a43ca4c43ab91f555433ca24ba2" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:51.104966 12717 raft_consensus.cc:2454] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:51.105026 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a802a43ca4c43ab91f555433ca24ba2" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:51.105167 12979 raft_consensus.cc:2454] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:51.105479 14893 leader_election.cc:302] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:51.105656 20315 raft_consensus.cc:2781] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.105715 20315 raft_consensus.cc:482] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.105759 20315 raft_consensus.cc:3037] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.106879 20315 raft_consensus.cc:504] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.107257 20315 leader_election.cc:288] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:51.107546 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a802a43ca4c43ab91f555433ca24ba2" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:51.107653 12979 raft_consensus.cc:3037] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.107689 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a802a43ca4c43ab91f555433ca24ba2" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:51.107853 12717 raft_consensus.cc:3037] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.108588 12979 raft_consensus.cc:2454] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:51.108836 12717 raft_consensus.cc:2454] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:51.109144 14893 leader_election.cc:302] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:51.109341 20315 raft_consensus.cc:2781] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.109447 20315 raft_consensus.cc:686] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:51.109601 20315 consensus_queue.cc:237] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.110342 20489 consensus_queue.cc:1035] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.112275 12441 catalog_manager.cc:5484] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:51.116511 20315 consensus_queue.cc:1035] T fd2686b167a941a4ae94881663197015 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.155738 19584 consensus_queue.cc:1035] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.158891 20226 consensus_queue.cc:1035] T 628137adc39c4ce29342326a171c30d7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:51.180330 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:51.186957 12765 tablet.cc:2369] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:09:51.222687 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:51.232070 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:51.236593 12717 raft_consensus.cc:1261] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.236622 12978 raft_consensus.cc:1261] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.237154 20315 consensus_queue.cc:1035] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.237426 20315 consensus_queue.cc:1035] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.239810 20447 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.240154 20447 tablet.cc:1620] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:51.240370 20247 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.240628 20247 tablet.cc:1620] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:51.243999 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=9ecef44c13b447fc900641841726f100] alter complete (version 1)
I20240501 14:09:51.245954 20510 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.246663 20510 tablet.cc:1620] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:51.247701 12439 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet 2a802a43ca4c43ab91f555433ca24ba2 (table rebalancer_dynamic_table [id=9ecef44c13b447fc900641841726f100]). Expected version 1 got 0
I20240501 14:09:51.264232 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:51.270646 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 4d446e4ed2a44be4bd565dbaebb29c1a (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.271049 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d446e4ed2a44be4bd565dbaebb29c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.271636 14937 tablet_service.cc:1467] Processing CreateTablet for tablet c1a1d1c489094aa9ac8cb87222067dc7 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.271879 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a1d1c489094aa9ac8cb87222067dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.271802 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 4d446e4ed2a44be4bd565dbaebb29c1a (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.272135 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d446e4ed2a44be4bd565dbaebb29c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.273427 20484 tablet_bootstrap.cc:492] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:51.273654 12959 tablet_service.cc:1467] Processing CreateTablet for tablet c1a1d1c489094aa9ac8cb87222067dc7 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.273681 12829 tablet_service.cc:1467] Processing CreateTablet for tablet ae91adb4afae4197be993d6050435d1d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.273850 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a1d1c489094aa9ac8cb87222067dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.273908 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae91adb4afae4197be993d6050435d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.274569 15202 tablet_service.cc:1467] Processing CreateTablet for tablet ae91adb4afae4197be993d6050435d1d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.274727 20484 tablet_bootstrap.cc:654] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.274905 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae91adb4afae4197be993d6050435d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.275959 12565 tablet_service.cc:1467] Processing CreateTablet for tablet c1a1d1c489094aa9ac8cb87222067dc7 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.276266 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1a1d1c489094aa9ac8cb87222067dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.276876 20483 tablet_bootstrap.cc:492] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:51.277240 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 4d446e4ed2a44be4bd565dbaebb29c1a (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.277463 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d446e4ed2a44be4bd565dbaebb29c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.277837 20483 tablet_bootstrap.cc:654] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.278769 20514 tablet_bootstrap.cc:492] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:51.278987 12567 tablet_service.cc:1467] Processing CreateTablet for tablet ae91adb4afae4197be993d6050435d1d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=d02b5dfc603d4fa18b63b936c108b35e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.279220 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae91adb4afae4197be993d6050435d1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.279677 20484 tablet_bootstrap.cc:492] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:51.279757 20484 ts_tablet_manager.cc:1367] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:51.279980 20514 tablet_bootstrap.cc:654] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.280404 20484 raft_consensus.cc:348] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.280516 20484 raft_consensus.cc:374] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.280548 20484 raft_consensus.cc:729] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.280683 20484 consensus_queue.cc:260] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.281003 20484 ts_tablet_manager.cc:1398] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.281188 20484 tablet_bootstrap.cc:492] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:51.281248 20513 tablet_bootstrap.cc:492] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:51.281471 20483 tablet_bootstrap.cc:492] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:51.281549 20483 ts_tablet_manager.cc:1367] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:51.282114 20484 tablet_bootstrap.cc:654] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.282071 20483 raft_consensus.cc:348] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.282178 20483 raft_consensus.cc:374] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.282225 20483 raft_consensus.cc:729] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.282337 20483 consensus_queue.cc:260] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.282636 20483 ts_tablet_manager.cc:1398] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.283061 20512 tablet_bootstrap.cc:492] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:51.284117 20512 tablet_bootstrap.cc:654] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.285277 20513 tablet_bootstrap.cc:654] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.286135 20512 tablet_bootstrap.cc:492] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:51.286278 20512 ts_tablet_manager.cc:1367] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:51.286396 20484 tablet_bootstrap.cc:492] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:51.286477 20484 ts_tablet_manager.cc:1367] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:51.286814 20512 raft_consensus.cc:348] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.286929 20512 raft_consensus.cc:374] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.286960 20484 raft_consensus.cc:348] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.287065 20484 raft_consensus.cc:374] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.287071 20512 raft_consensus.cc:729] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.287096 20484 raft_consensus.cc:729] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.287221 20484 consensus_queue.cc:260] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.287248 20512 consensus_queue.cc:260] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.287349 20513 tablet_bootstrap.cc:492] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:51.287422 20513 ts_tablet_manager.cc:1367] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:51.287508 20484 ts_tablet_manager.cc:1398] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.287532 20512 ts_tablet_manager.cc:1398] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.287907 20513 raft_consensus.cc:348] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.288012 20513 raft_consensus.cc:374] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.288064 20513 raft_consensus.cc:729] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.288203 20513 consensus_queue.cc:260] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.288547 20513 ts_tablet_manager.cc:1398] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.289072 20514 tablet_bootstrap.cc:492] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:51.289149 20514 ts_tablet_manager.cc:1367] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:09:51.289713 20513 tablet_bootstrap.cc:492] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:51.289638 20514 raft_consensus.cc:348] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.289760 20514 raft_consensus.cc:374] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.290650 20514 raft_consensus.cc:729] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.290889 20514 consensus_queue.cc:260] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.291287 20513 tablet_bootstrap.cc:654] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
W20240501 14:09:51.291626 13027 tablet.cc:2369] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:51.292042 20514 ts_tablet_manager.cc:1398] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:51.293784 20514 tablet_bootstrap.cc:492] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:51.294298 20152 raft_consensus.cc:482] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.294390 20152 raft_consensus.cc:504] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.294929 20514 tablet_bootstrap.cc:654] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.294945 20152 leader_election.cc:288] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:51.295181 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae91adb4afae4197be993d6050435d1d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:51.295312 12849 raft_consensus.cc:2454] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:51.295362 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae91adb4afae4197be993d6050435d1d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:51.295605 15157 leader_election.cc:302] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
W20240501 14:09:51.295699 15156 leader_election.cc:341] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:09:51.295776 20152 raft_consensus.cc:2781] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.295819 20152 raft_consensus.cc:482] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.295881 20152 raft_consensus.cc:3037] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.296613 20514 tablet_bootstrap.cc:492] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:51.296689 20514 ts_tablet_manager.cc:1367] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:51.296835 20152 raft_consensus.cc:504] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.297101 20152 leader_election.cc:288] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:51.297091 20514 raft_consensus.cc:348] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.297194 20514 raft_consensus.cc:374] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.297227 20514 raft_consensus.cc:729] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.297323 20514 consensus_queue.cc:260] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.297389 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae91adb4afae4197be993d6050435d1d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:51.297467 12585 raft_consensus.cc:3037] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.297631 20514 ts_tablet_manager.cc:1398] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.297770 20514 tablet_bootstrap.cc:492] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:51.298441 12585 raft_consensus.cc:2454] T ae91adb4afae4197be993d6050435d1d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:51.298696 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae91adb4afae4197be993d6050435d1d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:51.298871 15156 leader_election.cc:302] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:51.298902 12849 raft_consensus.cc:3037] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.299113 20152 raft_consensus.cc:2781] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.299191 20152 raft_consensus.cc:686] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:51.299396 20152 consensus_queue.cc:237] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.300595 20514 tablet_bootstrap.cc:654] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.300796 12849 raft_consensus.cc:2454] T ae91adb4afae4197be993d6050435d1d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:51.302661 20513 tablet_bootstrap.cc:492] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:51.302903 20513 ts_tablet_manager.cc:1367] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240501 14:09:51.306442 20513 raft_consensus.cc:348] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.306581 20513 raft_consensus.cc:374] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.306634 20513 raft_consensus.cc:729] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.306784 20513 consensus_queue.cc:260] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.307143 20514 tablet_bootstrap.cc:492] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:51.307154 20513 ts_tablet_manager.cc:1398] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:09:51.307240 20514 ts_tablet_manager.cc:1367] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240501 14:09:51.307492 12439 catalog_manager.cc:5484] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:51.307631 20514 raft_consensus.cc:348] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.307736 20514 raft_consensus.cc:374] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.307762 20514 raft_consensus.cc:729] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.307891 20514 consensus_queue.cc:260] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.308161 20514 ts_tablet_manager.cc:1398] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240501 14:09:51.344662 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:51.348177 20311 raft_consensus.cc:482] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.348277 20311 raft_consensus.cc:504] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.348554 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:51.348670 20311 leader_election.cc:288] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:51.348965 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d446e4ed2a44be4bd565dbaebb29c1a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:51.349105 12849 raft_consensus.cc:2454] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:51.349399 12916 leader_election.cc:302] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:51.349443 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d446e4ed2a44be4bd565dbaebb29c1a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:51.349588 20311 raft_consensus.cc:2781] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.349591 12585 raft_consensus.cc:2454] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:51.349646 20311 raft_consensus.cc:482] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.349686 20311 raft_consensus.cc:3037] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.350872 20311 raft_consensus.cc:504] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.351238 20311 leader_election.cc:288] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:51.351528 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d446e4ed2a44be4bd565dbaebb29c1a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:51.351593 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d446e4ed2a44be4bd565dbaebb29c1a" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:51.351630 12849 raft_consensus.cc:3037] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.351683 12585 raft_consensus.cc:3037] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.352367 12585 raft_consensus.cc:2454] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:51.352656 12849 raft_consensus.cc:2454] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:51.352931 12916 leader_election.cc:302] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:51.353123 20311 raft_consensus.cc:2781] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.353197 20311 raft_consensus.cc:686] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:51.353345 20311 consensus_queue.cc:237] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:51.355005 20447 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:51.355360 20447 tablet.cc:1620] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:51.355701 20247 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:51.356000 20247 tablet.cc:1620] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:51.356460 12440 catalog_manager.cc:5484] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:51.357936 20499 raft_consensus.cc:482] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.357897 20510 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:51.358036 20499 raft_consensus.cc:504] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.358178 20510 tablet.cc:1620] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:51.358520 20499 leader_election.cc:288] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:51.359966 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a1d1c489094aa9ac8cb87222067dc7" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:51.360108 14957 raft_consensus.cc:2454] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:51.360409 12521 leader_election.cc:302] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:51.360890 20499 raft_consensus.cc:2781] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.360968 20499 raft_consensus.cc:482] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.361008 20499 raft_consensus.cc:3037] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.362394 20499 raft_consensus.cc:504] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.362725 20499 leader_election.cc:288] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:51.363610 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a1d1c489094aa9ac8cb87222067dc7" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:51.363723 14957 raft_consensus.cc:3037] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.364403 14957 raft_consensus.cc:2454] T c1a1d1c489094aa9ac8cb87222067dc7 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:51.365129 12521 leader_election.cc:302] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:51.365396 20190 raft_consensus.cc:2781] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.365607 20190 raft_consensus.cc:686] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:51.365753 20190 consensus_queue.cc:237] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:51.366111 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a1d1c489094aa9ac8cb87222067dc7" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:51.366228 12978 raft_consensus.cc:3037] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.366950 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=9ecef44c13b447fc900641841726f100] alter complete (version 2)
I20240501 14:09:51.367312 12978 raft_consensus.cc:2454] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:51.368750 12443 catalog_manager.cc:5484] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:51.369267 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:51.369560 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1a1d1c489094aa9ac8cb87222067dc7" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:51.369715 12979 raft_consensus.cc:2362] T c1a1d1c489094aa9ac8cb87222067dc7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1. Re-sending same reply.
W20240501 14:09:51.375495 12896 tablet.cc:2369] T 4d446e4ed2a44be4bd565dbaebb29c1a P 4902d2609e6d44d089590a42e70356b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:51.471341 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:51.476692 20447 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:51.477018 20247 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:51.480353 20510 tablet.cc:1722] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:51.482802 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=9ecef44c13b447fc900641841726f100] alter complete (version 3)
W20240501 14:09:51.500504 12634 tablet.cc:2369] T 4d446e4ed2a44be4bd565dbaebb29c1a P 6a3cc0660eed427eaee38164a8c1d4c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:09:51.587085 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:51.587239 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:51.592427 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 2a802a43ca4c43ab91f555433ca24ba2
W20240501 14:09:51.592419 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:51.593118 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a802a43ca4c43ab91f555433ca24ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:51 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:51.593135 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a802a43ca4c43ab91f555433ca24ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:51 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:51.593178 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a802a43ca4c43ab91f555433ca24ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:51 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:51.593675 20549 tablet_replica.cc:330] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:51.593827 20549 raft_consensus.cc:2229] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:51.594020 20549 raft_consensus.cc:2258] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.594245 20550 tablet_replica.cc:330] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:51.594412 20550 raft_consensus.cc:2229] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:51.594589 20550 raft_consensus.cc:2258] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.594674 20549 ts_tablet_manager.cc:1875] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:51.595275 20550 ts_tablet_manager.cc:1875] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:51.597075 20549 ts_tablet_manager.cc:1888] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:51.597172 20549 log.cc:1196] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/2a802a43ca4c43ab91f555433ca24ba2
I20240501 14:09:51.597452 20549 ts_tablet_manager.cc:1909] T 2a802a43ca4c43ab91f555433ca24ba2 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:51.597877 20550 ts_tablet_manager.cc:1888] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:51.598024 20550 log.cc:1196] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/2a802a43ca4c43ab91f555433ca24ba2
I20240501 14:09:51.598388 20550 ts_tablet_manager.cc:1909] T 2a802a43ca4c43ab91f555433ca24ba2 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:51.598845 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 2a802a43ca4c43ab91f555433ca24ba2 (table rebalancer_dynamic_table_new_name [id=9ecef44c13b447fc900641841726f100]) successfully deleted
I20240501 14:09:51.599128 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 2a802a43ca4c43ab91f555433ca24ba2 (table rebalancer_dynamic_table_new_name [id=9ecef44c13b447fc900641841726f100]) successfully deleted
I20240501 14:09:51.602861 20551 tablet_replica.cc:330] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:51.603000 20551 raft_consensus.cc:2229] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:51.603127 20551 raft_consensus.cc:2258] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:51.603710 20551 ts_tablet_manager.cc:1875] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:51.605621 20551 ts_tablet_manager.cc:1888] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:51.605690 20551 log.cc:1196] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a802a43ca4c43ab91f555433ca24ba2
I20240501 14:09:51.605978 20551 ts_tablet_manager.cc:1909] T 2a802a43ca4c43ab91f555433ca24ba2 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:51.606714 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 2a802a43ca4c43ab91f555433ca24ba2 (table rebalancer_dynamic_table_new_name [id=9ecef44c13b447fc900641841726f100]) successfully deleted
I20240501 14:09:51.694602 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:51.700660 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 69327cc975f94ec481d8e9b151602300 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6]), partition=
I20240501 14:09:51.700765 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 69327cc975f94ec481d8e9b151602300 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6]), partition=
I20240501 14:09:51.700942 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69327cc975f94ec481d8e9b151602300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.700909 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 69327cc975f94ec481d8e9b151602300 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6]), partition=
I20240501 14:09:51.701059 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69327cc975f94ec481d8e9b151602300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.701120 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69327cc975f94ec481d8e9b151602300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.703024 20552 tablet_bootstrap.cc:492] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:51.703229 20483 tablet_bootstrap.cc:492] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:51.703429 20553 tablet_bootstrap.cc:492] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:51.704046 20552 tablet_bootstrap.cc:654] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.704190 20483 tablet_bootstrap.cc:654] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.704300 20553 tablet_bootstrap.cc:654] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.705251 20552 tablet_bootstrap.cc:492] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:51.705343 20552 ts_tablet_manager.cc:1367] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.705727 20553 tablet_bootstrap.cc:492] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:51.705850 20553 ts_tablet_manager.cc:1367] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:51.705865 20552 raft_consensus.cc:348] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.705994 20552 raft_consensus.cc:374] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.706034 20552 raft_consensus.cc:729] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.706159 20552 consensus_queue.cc:260] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.706255 20553 raft_consensus.cc:348] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.706389 20553 raft_consensus.cc:374] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.706432 20552 ts_tablet_manager.cc:1398] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.706456 20553 raft_consensus.cc:729] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.706581 20553 consensus_queue.cc:260] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.706876 20553 ts_tablet_manager.cc:1398] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.707166 20483 tablet_bootstrap.cc:492] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:51.707263 20483 ts_tablet_manager.cc:1367] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:51.707731 20483 raft_consensus.cc:348] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.707825 20483 raft_consensus.cc:374] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.707854 20483 raft_consensus.cc:729] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.707971 20483 consensus_queue.cc:260] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.708250 20483 ts_tablet_manager.cc:1398] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:51.720719 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:51.774931 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:51.785004 20529 consensus_queue.cc:1035] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.789036 20152 consensus_queue.cc:1035] T ae91adb4afae4197be993d6050435d1d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.819197 20536 consensus_queue.cc:1035] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.822885 20190 consensus_queue.cc:1035] T c1a1d1c489094aa9ac8cb87222067dc7 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.832404 20489 raft_consensus.cc:482] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.832515 20489 raft_consensus.cc:504] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.832937 20489 leader_election.cc:288] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:51.833273 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69327cc975f94ec481d8e9b151602300" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:51.833297 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69327cc975f94ec481d8e9b151602300" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:51.833389 15352 raft_consensus.cc:2454] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:51.833431 12717 raft_consensus.cc:2454] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:51.833689 14891 leader_election.cc:302] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:51.833866 20489 raft_consensus.cc:2781] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:51.833925 20489 raft_consensus.cc:482] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:51.833951 20489 raft_consensus.cc:3037] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.834947 20489 raft_consensus.cc:504] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.835302 20489 leader_election.cc:288] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:51.835675 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69327cc975f94ec481d8e9b151602300" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:51.835702 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69327cc975f94ec481d8e9b151602300" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:51.835808 15352 raft_consensus.cc:3037] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.835824 12717 raft_consensus.cc:3037] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:51.836583 15352 raft_consensus.cc:2454] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:51.836614 12717 raft_consensus.cc:2454] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:51.836920 14894 leader_election.cc:302] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:51.837102 20489 raft_consensus.cc:2781] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:51.837198 20489 raft_consensus.cc:686] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:51.837347 20489 consensus_queue.cc:237] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.839610 12443 catalog_manager.cc:5484] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:51.863370 20311 consensus_queue.cc:1035] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.867748 20311 consensus_queue.cc:1035] T 4d446e4ed2a44be4bd565dbaebb29c1a P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:51.881796 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:51.888460 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:51.894639 15332 tablet_service.cc:1467] Processing CreateTablet for tablet e9ff2b9229c84baa8cbc433f08d8ff3e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.895053 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ff2b9229c84baa8cbc433f08d8ff3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.895139 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5b677b79c44973bdcb7a6931d04084 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.895370 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5b677b79c44973bdcb7a6931d04084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.895511 14937 tablet_service.cc:1467] Processing CreateTablet for tablet e9ff2b9229c84baa8cbc433f08d8ff3e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.895763 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ff2b9229c84baa8cbc433f08d8ff3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.897266 20553 tablet_bootstrap.cc:492] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:51.897976 20483 tablet_bootstrap.cc:492] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:51.898272 20553 tablet_bootstrap.cc:654] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.899060 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 650e2f227ad64e97bd3967dea892caf7 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.899178 20483 tablet_bootstrap.cc:654] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.899183 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5b677b79c44973bdcb7a6931d04084 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.899346 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650e2f227ad64e97bd3967dea892caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.899398 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5b677b79c44973bdcb7a6931d04084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.899963 12696 tablet_service.cc:1467] Processing CreateTablet for tablet e9ff2b9229c84baa8cbc433f08d8ff3e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.900182 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ff2b9229c84baa8cbc433f08d8ff3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.901160 20581 tablet_bootstrap.cc:492] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:51.901588 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 650e2f227ad64e97bd3967dea892caf7 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.901831 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650e2f227ad64e97bd3967dea892caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.902134 20553 tablet_bootstrap.cc:492] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:51.902215 20553 ts_tablet_manager.cc:1367] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:51.902761 20581 tablet_bootstrap.cc:654] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.902695 20553 raft_consensus.cc:348] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.902817 20553 raft_consensus.cc:374] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.902851 20553 raft_consensus.cc:729] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.902868 20552 tablet_bootstrap.cc:492] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:51.902980 20553 consensus_queue.cc:260] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.903354 20553 ts_tablet_manager.cc:1398] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.903795 20552 tablet_bootstrap.cc:654] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.904193 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5b677b79c44973bdcb7a6931d04084 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.904417 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5b677b79c44973bdcb7a6931d04084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.904965 20483 tablet_bootstrap.cc:492] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:51.905063 20483 ts_tablet_manager.cc:1367] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:51.905540 20483 raft_consensus.cc:348] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.905649 20483 raft_consensus.cc:374] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.905685 20483 raft_consensus.cc:729] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.905793 20483 consensus_queue.cc:260] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.906118 20483 ts_tablet_manager.cc:1398] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.908283 20553 tablet_bootstrap.cc:492] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:51.909036 20553 tablet_bootstrap.cc:654] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.910776 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 650e2f227ad64e97bd3967dea892caf7 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0e1d70fecad94acf8eae2792d8824983]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:51.911098 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 650e2f227ad64e97bd3967dea892caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:51.911654 20553 tablet_bootstrap.cc:492] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:51.911783 20553 ts_tablet_manager.cc:1367] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:51.912171 20553 raft_consensus.cc:348] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:51.912258 20553 raft_consensus.cc:374] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.912294 20553 raft_consensus.cc:729] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.912443 20553 consensus_queue.cc:260] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:51.912799 20553 ts_tablet_manager.cc:1398] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.913465 20581 tablet_bootstrap.cc:492] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:51.913600 20581 ts_tablet_manager.cc:1367] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:51.914048 20581 raft_consensus.cc:348] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:51.914247 20581 raft_consensus.cc:374] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.914323 20581 raft_consensus.cc:729] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.915040 20552 tablet_bootstrap.cc:492] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:51.915146 20552 ts_tablet_manager.cc:1367] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:51.915087 20581 consensus_queue.cc:260] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:51.915377 20581 ts_tablet_manager.cc:1398] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.915540 20581 tablet_bootstrap.cc:492] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:51.915609 20552 raft_consensus.cc:348] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.915711 20552 raft_consensus.cc:374] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.915742 20552 raft_consensus.cc:729] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.915858 20552 consensus_queue.cc:260] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.916180 20552 ts_tablet_manager.cc:1398] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.916421 20552 tablet_bootstrap.cc:492] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:51.916568 20578 tablet_bootstrap.cc:492] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:51.916783 20581 tablet_bootstrap.cc:654] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.917479 20552 tablet_bootstrap.cc:654] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.917799 20581 tablet_bootstrap.cc:492] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:51.917865 20581 ts_tablet_manager.cc:1367] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:51.918165 20578 tablet_bootstrap.cc:654] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.918295 20581 raft_consensus.cc:348] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.918398 20581 raft_consensus.cc:374] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.918435 20581 raft_consensus.cc:729] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.918565 20581 consensus_queue.cc:260] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.918934 20581 ts_tablet_manager.cc:1398] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:51.920545 20578 tablet_bootstrap.cc:492] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:51.920629 20578 ts_tablet_manager.cc:1367] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:51.920986 20578 raft_consensus.cc:348] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:51.921097 20578 raft_consensus.cc:374] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.921133 20578 raft_consensus.cc:729] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.921226 20578 consensus_queue.cc:260] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:51.921556 20578 ts_tablet_manager.cc:1398] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:51.921700 20578 tablet_bootstrap.cc:492] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:51.922539 20578 tablet_bootstrap.cc:654] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:51.924000 20552 tablet_bootstrap.cc:492] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:51.924096 20552 ts_tablet_manager.cc:1367] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
W20240501 14:09:51.924479 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:51.924737 20552 raft_consensus.cc:348] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.925194 20552 raft_consensus.cc:374] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.925238 20552 raft_consensus.cc:729] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.925354 20552 consensus_queue.cc:260] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:51.926262 20552 ts_tablet_manager.cc:1398] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:51.926990 20578 tablet_bootstrap.cc:492] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:51.927116 20578 ts_tablet_manager.cc:1367] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:51.927628 20578 raft_consensus.cc:348] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.927702 20578 raft_consensus.cc:374] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:51.927721 20578 raft_consensus.cc:729] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:51.927816 20578 consensus_queue.cc:260] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:51.928062 20578 ts_tablet_manager.cc:1398] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:51.956063 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:51.960109 12717 raft_consensus.cc:1261] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.960189 15352 raft_consensus.cc:1261] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:51.960556 20315 consensus_queue.cc:1035] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.960682 20489 consensus_queue.cc:1035] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:51.964830 20447 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.965169 20447 tablet.cc:1620] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:51.966070 20593 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.966392 20593 tablet.cc:1620] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:51.966625 12439 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet 69327cc975f94ec481d8e9b151602300 (table rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6]). Expected version 1 got 0
I20240501 14:09:51.970633 12441 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6] alter complete (version 1)
I20240501 14:09:51.972043 20510 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:51.972354 20510 tablet.cc:1620] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:51.975539 12443 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet 69327cc975f94ec481d8e9b151602300 (table rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6]). Expected version 1 got 0
I20240501 14:09:52.025985 20582 raft_consensus.cc:482] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.026113 20582 raft_consensus.cc:504] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.026576 20582 leader_election.cc:288] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:52.026955 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5b677b79c44973bdcb7a6931d04084" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:52.026981 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5b677b79c44973bdcb7a6931d04084" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:52.027084 12979 raft_consensus.cc:2454] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:52.027115 15222 raft_consensus.cc:2454] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:52.027386 15288 leader_election.cc:302] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:52.027557 20276 raft_consensus.cc:2781] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.027614 20276 raft_consensus.cc:482] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.027660 20276 raft_consensus.cc:3037] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.028595 20276 raft_consensus.cc:504] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.028914 20276 leader_election.cc:288] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:52.029271 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5b677b79c44973bdcb7a6931d04084" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:52.029309 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5b677b79c44973bdcb7a6931d04084" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:52.029384 12979 raft_consensus.cc:3037] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.029436 15222 raft_consensus.cc:3037] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.029742 20489 raft_consensus.cc:482] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.029845 20489 raft_consensus.cc:504] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:52.030241 20489 leader_election.cc:288] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:52.030408 15222 raft_consensus.cc:2454] T 4e5b677b79c44973bdcb7a6931d04084 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:52.030426 12979 raft_consensus.cc:2454] T 4e5b677b79c44973bdcb7a6931d04084 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:52.030601 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff2b9229c84baa8cbc433f08d8ff3e" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:52.030691 15286 leader_election.cc:302] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:52.030719 15352 raft_consensus.cc:2454] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:52.030946 20276 raft_consensus.cc:2781] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.031015 14894 leader_election.cc:302] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:52.031045 20276 raft_consensus.cc:686] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:52.031173 20489 raft_consensus.cc:2781] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.031234 20489 raft_consensus.cc:482] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.031181 20276 consensus_queue.cc:237] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.031277 20489 raft_consensus.cc:3037] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.032187 20489 raft_consensus.cc:504] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:52.032536 20489 leader_election.cc:288] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:52.033020 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff2b9229c84baa8cbc433f08d8ff3e" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:52.033084 19584 raft_consensus.cc:482] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.033154 19584 raft_consensus.cc:504] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:52.033315 12717 raft_consensus.cc:2454] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:52.033582 19584 leader_election.cc:288] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:52.033614 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff2b9229c84baa8cbc433f08d8ff3e" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:52.033720 15352 raft_consensus.cc:3037] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.033819 12443 catalog_manager.cc:5484] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:52.033944 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff2b9229c84baa8cbc433f08d8ff3e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:52.033944 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff2b9229c84baa8cbc433f08d8ff3e" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:52.034085 14957 raft_consensus.cc:2379] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in current term 1: Already voted for candidate 8a13c028ba8a4a659b74af6c265df95e in this term.
I20240501 14:09:52.034391 12652 leader_election.cc:302] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8
I20240501 14:09:52.034488 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9ff2b9229c84baa8cbc433f08d8ff3e" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:52.034564 19584 raft_consensus.cc:3037] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.034632 15352 raft_consensus.cc:2454] T e9ff2b9229c84baa8cbc433f08d8ff3e P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:52.035456 19584 raft_consensus.cc:2726] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:09:52.035555 14894 leader_election.cc:302] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:52.035729 20489 raft_consensus.cc:2781] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.035799 20489 raft_consensus.cc:686] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:52.035912 20489 consensus_queue.cc:237] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:52.036520 12716 raft_consensus.cc:2454] T e9ff2b9229c84baa8cbc433f08d8ff3e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:52.039497 12443 catalog_manager.cc:5484] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:52.043699 20529 raft_consensus.cc:482] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.043804 20529 raft_consensus.cc:504] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.044256 20529 leader_election.cc:288] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:52.044553 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650e2f227ad64e97bd3967dea892caf7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:52.044581 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650e2f227ad64e97bd3967dea892caf7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:52.044693 12716 raft_consensus.cc:2454] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:52.044701 12979 raft_consensus.cc:2454] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:52.045030 15154 leader_election.cc:302] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:52.045526 20529 raft_consensus.cc:2781] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.045605 20529 raft_consensus.cc:482] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.045641 20529 raft_consensus.cc:3037] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.046689 20529 raft_consensus.cc:504] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.047032 20529 leader_election.cc:288] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:52.047354 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650e2f227ad64e97bd3967dea892caf7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:52.047479 12716 raft_consensus.cc:3037] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.048661 12716 raft_consensus.cc:2454] T 650e2f227ad64e97bd3967dea892caf7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:52.048897 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "650e2f227ad64e97bd3967dea892caf7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:52.049012 12979 raft_consensus.cc:3037] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.049019 15154 leader_election.cc:302] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:52.049901 12979 raft_consensus.cc:2454] T 650e2f227ad64e97bd3967dea892caf7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
W20240501 14:09:52.050256 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.051127 20529 raft_consensus.cc:2781] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.051218 20529 raft_consensus.cc:686] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:52.051345 20529 consensus_queue.cc:237] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.053951 12443 catalog_manager.cc:5484] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:52.075287 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240501 14:09:52.080462 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:52.080560 20447 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.080876 20447 tablet.cc:1620] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:52.081785 20510 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.082079 20510 tablet.cc:1620] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:52.083657 20593 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.083949 20593 tablet.cc:1620] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:52.087795 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=475d45831ef44180bc1fb44e878c72f6] alter complete (version 2)
W20240501 14:09:52.103721 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:52.166008 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:52.192498 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:52.197990 20447 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.198367 20510 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.200701 20593 tablet.cc:1722] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 14:09:52.201061 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.205418 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=475d45831ef44180bc1fb44e878c72f6] alter complete (version 3)
W20240501 14:09:52.208123 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:09:52.208848 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:52.265852 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:09:52.285562 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:09:52.290858 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:09:52.293836 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:09:52.294477 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.306876 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:52.307026 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:52.309087 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 69327cc975f94ec481d8e9b151602300
I20240501 14:09:52.309662 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 69327cc975f94ec481d8e9b151602300 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:52.309675 15332 tablet_service.cc:1514] Processing DeleteTablet for tablet 69327cc975f94ec481d8e9b151602300 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:52.309863 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 69327cc975f94ec481d8e9b151602300 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:52.309978 20619 tablet_replica.cc:330] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:52.310091 20620 tablet_replica.cc:330] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:52.310160 20619 raft_consensus.cc:2229] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:52.310256 20620 raft_consensus.cc:2229] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.310281 20621 tablet_replica.cc:330] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:52.310391 20619 raft_consensus.cc:2258] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.310412 20620 raft_consensus.cc:2258] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.310424 20621 raft_consensus.cc:2229] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.311075 20619 ts_tablet_manager.cc:1875] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.311441 20621 raft_consensus.cc:2258] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.311213 20620 ts_tablet_manager.cc:1875] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.313118 20621 ts_tablet_manager.cc:1875] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.313556 20619 ts_tablet_manager.cc:1888] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.313644 20619 log.cc:1196] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/69327cc975f94ec481d8e9b151602300
I20240501 14:09:52.313912 20619 ts_tablet_manager.cc:1909] T 69327cc975f94ec481d8e9b151602300 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:52.313916 20620 ts_tablet_manager.cc:1888] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.313977 20620 log.cc:1196] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/69327cc975f94ec481d8e9b151602300
I20240501 14:09:52.314255 20620 ts_tablet_manager.cc:1909] T 69327cc975f94ec481d8e9b151602300 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:52.314718 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 69327cc975f94ec481d8e9b151602300 (table rebalancer_dynamic_table_new_name [id=475d45831ef44180bc1fb44e878c72f6]) successfully deleted
I20240501 14:09:52.315167 20621 ts_tablet_manager.cc:1888] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.315246 20621 log.cc:1196] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/69327cc975f94ec481d8e9b151602300
I20240501 14:09:52.315428 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet 69327cc975f94ec481d8e9b151602300 (table rebalancer_dynamic_table_new_name [id=475d45831ef44180bc1fb44e878c72f6]) successfully deleted
I20240501 14:09:52.315534 20621 ts_tablet_manager.cc:1909] T 69327cc975f94ec481d8e9b151602300 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
W20240501 14:09:52.316188 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:09:52.316299 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 69327cc975f94ec481d8e9b151602300 (table rebalancer_dynamic_table_new_name [id=475d45831ef44180bc1fb44e878c72f6]) successfully deleted
I20240501 14:09:52.411098 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:52.419250 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 0688c124ebc34ed7974a90c4cae4efd8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a]), partition=
I20240501 14:09:52.419567 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0688c124ebc34ed7974a90c4cae4efd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.419829 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 0688c124ebc34ed7974a90c4cae4efd8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a]), partition=
I20240501 14:09:52.420073 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0688c124ebc34ed7974a90c4cae4efd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.420091 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 0688c124ebc34ed7974a90c4cae4efd8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a]), partition=
I20240501 14:09:52.420325 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0688c124ebc34ed7974a90c4cae4efd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.422457 20628 tablet_bootstrap.cc:492] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:52.423060 20630 tablet_bootstrap.cc:492] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:52.424654 20628 tablet_bootstrap.cc:654] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.424999 20629 tablet_bootstrap.cc:492] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:52.425982 20629 tablet_bootstrap.cc:654] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.426013 20630 tablet_bootstrap.cc:654] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.427588 20629 tablet_bootstrap.cc:492] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:52.427708 20629 ts_tablet_manager.cc:1367] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:52.427176 20628 tablet_bootstrap.cc:492] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:52.428248 20630 tablet_bootstrap.cc:492] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:52.428324 20628 ts_tablet_manager.cc:1367] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:52.428369 20630 ts_tablet_manager.cc:1367] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:52.428705 20629 raft_consensus.cc:348] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.428826 20628 raft_consensus.cc:348] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.428828 20630 raft_consensus.cc:348] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.428936 20630 raft_consensus.cc:374] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.428920 20629 raft_consensus.cc:374] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.428982 20629 raft_consensus.cc:729] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.428995 20630 raft_consensus.cc:729] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.429072 20628 raft_consensus.cc:374] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.429136 20630 consensus_queue.cc:260] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.429188 20628 raft_consensus.cc:729] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.429159 20629 consensus_queue.cc:260] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.429463 20630 ts_tablet_manager.cc:1398] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.429574 20629 ts_tablet_manager.cc:1398] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:52.430909 20628 consensus_queue.cc:260] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.431126 20190 raft_consensus.cc:482] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.431216 20190 raft_consensus.cc:504] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.431715 20190 leader_election.cc:288] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:52.432144 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0688c124ebc34ed7974a90c4cae4efd8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:52.432271 12848 raft_consensus.cc:2454] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:52.432343 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0688c124ebc34ed7974a90c4cae4efd8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:52.432452 14958 raft_consensus.cc:2454] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:52.432543 12523 leader_election.cc:302] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:52.432701 20190 raft_consensus.cc:2781] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.432754 20628 ts_tablet_manager.cc:1398] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:52.432770 20190 raft_consensus.cc:482] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.432817 20190 raft_consensus.cc:3037] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.433161 20582 consensus_queue.cc:1035] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.433946 20190 raft_consensus.cc:504] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.434288 20190 leader_election.cc:288] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:52.435108 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0688c124ebc34ed7974a90c4cae4efd8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:52.435207 12848 raft_consensus.cc:3037] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.436234 12848 raft_consensus.cc:2454] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:52.436429 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0688c124ebc34ed7974a90c4cae4efd8" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:52.436533 14958 raft_consensus.cc:3037] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.437356 14958 raft_consensus.cc:2454] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:52.437353 12523 leader_election.cc:302] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:52.437521 20190 raft_consensus.cc:2781] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.437610 20190 raft_consensus.cc:686] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:52.437741 20190 consensus_queue.cc:237] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.444695 12440 catalog_manager.cc:5484] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:52.444876 20276 consensus_queue.cc:1035] T 4e5b677b79c44973bdcb7a6931d04084 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:52.449648 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:09:52.451962 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:09:52.470081 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:52.483403 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:52.510823 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:52.532814 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:52.541607 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:52.542374 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.543823 20529 consensus_queue.cc:1035] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.548756 20152 consensus_queue.cc:1035] T 650e2f227ad64e97bd3967dea892caf7 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.549796 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:52.554018 12848 raft_consensus.cc:1261] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:52.554006 14958 raft_consensus.cc:1261] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:52.554454 20190 consensus_queue.cc:1035] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.555022 20190 consensus_queue.cc:1035] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.558111 12439 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 0688c124ebc34ed7974a90c4cae4efd8 (table rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a]). Expected version 1 got 0
I20240501 14:09:52.558866 20647 mvcc.cc:205] Tried to move back new op lower bound from 7022889339096494080 to 7022889338626248704. Current Snapshot: MvccSnapshot[applied={T|T < 7022889339096494080}]
I20240501 14:09:52.559309 20447 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:52.559566 20651 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:52.559636 20447 tablet.cc:1620] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:52.559864 20651 tablet.cc:1620] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:52.560523 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 0688c124ebc34ed7974a90c4cae4efd8 (table rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a]). Expected version 1 got 0
I20240501 14:09:52.560845 12440 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 0688c124ebc34ed7974a90c4cae4efd8 (table rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a]). Expected version 1 got 0
I20240501 14:09:52.561981 20650 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:52.562283 20650 tablet.cc:1620] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:52.565034 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:52.564874 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a] alter complete (version 1)
W20240501 14:09:52.569506 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.571326 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5c69ea0a594bcfb775f1fb884a286a (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.571628 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5c69ea0a594bcfb775f1fb884a286a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.572185 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5c69ea0a594bcfb775f1fb884a286a (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.572260 12827 tablet_service.cc:1467] Processing CreateTablet for tablet bd1b710faf264200a2394303e4195edf (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.572440 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5c69ea0a594bcfb775f1fb884a286a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.572539 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1b710faf264200a2394303e4195edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.573154 14937 tablet_service.cc:1467] Processing CreateTablet for tablet bd1b710faf264200a2394303e4195edf (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.573410 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1b710faf264200a2394303e4195edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.574081 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 92603ceaeb2d4e279507d27a579dc60f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.574357 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92603ceaeb2d4e279507d27a579dc60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.574865 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 92603ceaeb2d4e279507d27a579dc60f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.574973 12698 tablet_service.cc:1467] Processing CreateTablet for tablet bd1b710faf264200a2394303e4195edf (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.575073 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92603ceaeb2d4e279507d27a579dc60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.575234 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1b710faf264200a2394303e4195edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.575726 20629 tablet_bootstrap.cc:492] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:52.575960 20630 tablet_bootstrap.cc:492] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:52.576735 20629 tablet_bootstrap.cc:654] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.576856 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 1e5c69ea0a594bcfb775f1fb884a286a (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.576918 20630 tablet_bootstrap.cc:654] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.577090 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e5c69ea0a594bcfb775f1fb884a286a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.577262 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 92603ceaeb2d4e279507d27a579dc60f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=021e3c64edc14fa7a1dbfaa0ca1a996a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:52.577490 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92603ceaeb2d4e279507d27a579dc60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:52.577580 20628 tablet_bootstrap.cc:492] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:52.578208 20652 tablet_bootstrap.cc:492] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:52.578490 20628 tablet_bootstrap.cc:654] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.579207 20652 tablet_bootstrap.cc:654] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.580904 20652 tablet_bootstrap.cc:492] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:52.580998 20652 ts_tablet_manager.cc:1367] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:52.581130 20630 tablet_bootstrap.cc:492] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:52.581205 20630 ts_tablet_manager.cc:1367] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:52.581321 20655 tablet_bootstrap.cc:492] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:52.581466 20652 raft_consensus.cc:348] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.581565 20652 raft_consensus.cc:374] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.581595 20652 raft_consensus.cc:729] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.581621 20630 raft_consensus.cc:348] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.581743 20630 raft_consensus.cc:374] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.581708 20652 consensus_queue.cc:260] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.581773 20630 raft_consensus.cc:729] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.581884 20630 consensus_queue.cc:260] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.582018 20652 ts_tablet_manager.cc:1398] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.582178 20652 tablet_bootstrap.cc:492] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:52.582240 20630 ts_tablet_manager.cc:1398] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.583016 20628 tablet_bootstrap.cc:492] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:52.583039 20652 tablet_bootstrap.cc:654] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.583109 20628 ts_tablet_manager.cc:1367] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:52.583537 20628 raft_consensus.cc:348] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.583657 20628 raft_consensus.cc:374] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.583684 20628 raft_consensus.cc:729] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.583787 20628 consensus_queue.cc:260] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.584081 20628 ts_tablet_manager.cc:1398] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.586673 20489 consensus_queue.cc:1035] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:52.587201 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.588564 20629 tablet_bootstrap.cc:492] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:52.588651 20629 ts_tablet_manager.cc:1367] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:52.589134 20629 raft_consensus.cc:348] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.589246 20629 raft_consensus.cc:374] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.589277 20629 raft_consensus.cc:729] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.589394 20629 consensus_queue.cc:260] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.589694 20629 ts_tablet_manager.cc:1398] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.590643 20489 consensus_queue.cc:1035] T e9ff2b9229c84baa8cbc433f08d8ff3e P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:52.595360 20652 tablet_bootstrap.cc:492] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:52.595467 20652 ts_tablet_manager.cc:1367] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240501 14:09:52.596092 20652 raft_consensus.cc:348] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.596206 20652 raft_consensus.cc:374] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.596248 20652 raft_consensus.cc:729] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.596367 20652 consensus_queue.cc:260] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.596675 20652 ts_tablet_manager.cc:1398] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:52.599177 20657 tablet_bootstrap.cc:492] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:52.599548 20655 tablet_bootstrap.cc:654] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.600226 20657 tablet_bootstrap.cc:654] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.600790 20655 tablet_bootstrap.cc:492] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:52.600883 20655 ts_tablet_manager.cc:1367] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:52.601387 20655 raft_consensus.cc:348] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.601536 20655 raft_consensus.cc:374] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.601603 20655 raft_consensus.cc:729] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.601775 20655 consensus_queue.cc:260] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.604534 20655 ts_tablet_manager.cc:1398] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:52.604718 20655 tablet_bootstrap.cc:492] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:52.605365 20657 tablet_bootstrap.cc:492] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:52.605450 20657 ts_tablet_manager.cc:1367] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:52.605875 20657 raft_consensus.cc:348] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.605981 20657 raft_consensus.cc:374] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.606019 20657 raft_consensus.cc:729] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.606184 20657 consensus_queue.cc:260] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.606470 20657 ts_tablet_manager.cc:1398] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.606638 20657 tablet_bootstrap.cc:492] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:52.608074 20657 tablet_bootstrap.cc:654] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.609380 20655 tablet_bootstrap.cc:654] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:52.610237 20657 tablet_bootstrap.cc:492] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:52.610324 20657 ts_tablet_manager.cc:1367] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:52.610828 20657 raft_consensus.cc:348] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.610993 20657 raft_consensus.cc:374] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.611091 20657 raft_consensus.cc:729] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.611335 20657 consensus_queue.cc:260] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.611758 20657 ts_tablet_manager.cc:1398] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:52.612452 20655 tablet_bootstrap.cc:492] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:52.612537 20655 ts_tablet_manager.cc:1367] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:52.612993 20655 raft_consensus.cc:348] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.613157 20655 raft_consensus.cc:374] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:52.613256 20655 raft_consensus.cc:729] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:52.613446 20655 consensus_queue.cc:260] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.613847 20655 ts_tablet_manager.cc:1398] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:52.644384 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:52.671574 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:52.677412 20650 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.677735 20650 tablet.cc:1620] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:52.677807 20651 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.677891 20447 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:52.678107 20651 tablet.cc:1620] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:52.678176 20447 tablet.cc:1620] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:52.683059 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=c3f9efb8422d4aee8a373f00bb7fad7a] alter complete (version 2)
I20240501 14:09:52.744246 20276 raft_consensus.cc:482] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.744382 20276 raft_consensus.cc:504] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.744763 20276 leader_election.cc:288] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:52.745164 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92603ceaeb2d4e279507d27a579dc60f" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:52.745164 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92603ceaeb2d4e279507d27a579dc60f" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:52.745309 15221 raft_consensus.cc:2454] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:52.745309 12716 raft_consensus.cc:2454] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:52.745627 15286 leader_election.cc:302] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:52.746012 20276 raft_consensus.cc:2781] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.746093 20276 raft_consensus.cc:482] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.746132 20276 raft_consensus.cc:3037] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:52.746614 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.747213 20276 raft_consensus.cc:504] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.747531 20276 leader_election.cc:288] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:52.747874 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92603ceaeb2d4e279507d27a579dc60f" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:52.747905 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92603ceaeb2d4e279507d27a579dc60f" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:52.748021 12716 raft_consensus.cc:3037] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.748036 15221 raft_consensus.cc:3037] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.748871 15221 raft_consensus.cc:2454] T 92603ceaeb2d4e279507d27a579dc60f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:52.748867 12716 raft_consensus.cc:2454] T 92603ceaeb2d4e279507d27a579dc60f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:52.749150 15286 leader_election.cc:302] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:52.749301 20276 raft_consensus.cc:2781] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.749501 20276 raft_consensus.cc:686] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:52.749635 20276 consensus_queue.cc:237] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:52.752418 12438 catalog_manager.cc:5484] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:52.784955 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:52.790392 20650 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.790762 20447 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.791599 20651 tablet.cc:1722] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:52.797216 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=c3f9efb8422d4aee8a373f00bb7fad7a] alter complete (version 3)
I20240501 14:09:52.866039 20152 raft_consensus.cc:482] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.866163 20152 raft_consensus.cc:504] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.866638 20152 leader_election.cc:288] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:52.867149 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5c69ea0a594bcfb775f1fb884a286a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:52.867154 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5c69ea0a594bcfb775f1fb884a286a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:52.867280 12585 raft_consensus.cc:2454] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:52.867296 15351 raft_consensus.cc:2454] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:52.867610 15156 leader_election.cc:302] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:52.868407 20152 raft_consensus.cc:2781] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.868470 20152 raft_consensus.cc:482] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.868515 20152 raft_consensus.cc:3037] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.870144 20152 raft_consensus.cc:504] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.870477 20152 leader_election.cc:288] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:52.871387 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5c69ea0a594bcfb775f1fb884a286a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:52.871407 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e5c69ea0a594bcfb775f1fb884a286a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:52.871524 15351 raft_consensus.cc:3037] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.871513 12585 raft_consensus.cc:3037] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.872368 12585 raft_consensus.cc:2454] T 1e5c69ea0a594bcfb775f1fb884a286a P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:52.872455 15351 raft_consensus.cc:2454] T 1e5c69ea0a594bcfb775f1fb884a286a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:52.872678 15156 leader_election.cc:302] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:52.873127 20152 raft_consensus.cc:2781] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.873263 20152 raft_consensus.cc:686] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:52.873396 20152 consensus_queue.cc:237] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:52.875648 12443 catalog_manager.cc:5484] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:52.898449 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:52.898624 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:52.900889 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 0688c124ebc34ed7974a90c4cae4efd8
I20240501 14:09:52.901492 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 0688c124ebc34ed7974a90c4cae4efd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:52.901492 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 0688c124ebc34ed7974a90c4cae4efd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:52.901630 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 0688c124ebc34ed7974a90c4cae4efd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:52 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:52.901844 20690 tablet_replica.cc:330] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:52.901974 20691 tablet_replica.cc:330] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:52.901985 20690 raft_consensus.cc:2229] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.902102 20690 raft_consensus.cc:2258] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.902134 20691 raft_consensus.cc:2229] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:52.902328 20689 tablet_replica.cc:330] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:52.902336 20691 raft_consensus.cc:2258] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.902470 20689 raft_consensus.cc:2229] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:52.902601 20689 raft_consensus.cc:2258] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:52.903069 20690 ts_tablet_manager.cc:1875] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.903069 20691 ts_tablet_manager.cc:1875] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.903221 20689 ts_tablet_manager.cc:1875] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:52.905457 20689 ts_tablet_manager.cc:1888] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.905522 20689 log.cc:1196] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/0688c124ebc34ed7974a90c4cae4efd8
I20240501 14:09:52.905508 20691 ts_tablet_manager.cc:1888] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.905539 20690 ts_tablet_manager.cc:1888] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:52.905578 20691 log.cc:1196] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/0688c124ebc34ed7974a90c4cae4efd8
I20240501 14:09:52.905599 20690 log.cc:1196] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/0688c124ebc34ed7974a90c4cae4efd8
I20240501 14:09:52.905804 20689 ts_tablet_manager.cc:1909] T 0688c124ebc34ed7974a90c4cae4efd8 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:52.905937 20690 ts_tablet_manager.cc:1909] T 0688c124ebc34ed7974a90c4cae4efd8 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:52.905937 20691 ts_tablet_manager.cc:1909] T 0688c124ebc34ed7974a90c4cae4efd8 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:52.906625 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 0688c124ebc34ed7974a90c4cae4efd8 (table rebalancer_dynamic_table_new_name [id=c3f9efb8422d4aee8a373f00bb7fad7a]) successfully deleted
I20240501 14:09:52.906651 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 0688c124ebc34ed7974a90c4cae4efd8 (table rebalancer_dynamic_table_new_name [id=c3f9efb8422d4aee8a373f00bb7fad7a]) successfully deleted
I20240501 14:09:52.906848 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 0688c124ebc34ed7974a90c4cae4efd8 (table rebalancer_dynamic_table_new_name [id=c3f9efb8422d4aee8a373f00bb7fad7a]) successfully deleted
W20240501 14:09:52.920424 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:52.929975 20315 raft_consensus.cc:482] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:52.930085 20315 raft_consensus.cc:504] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.930465 20315 leader_election.cc:288] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:52.931058 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1b710faf264200a2394303e4195edf" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:52.931056 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1b710faf264200a2394303e4195edf" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:52.931200 12716 raft_consensus.cc:2454] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:52.931211 12849 raft_consensus.cc:2454] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:52.931488 14894 leader_election.cc:302] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:52.931648 20315 raft_consensus.cc:2781] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:52.931711 20315 raft_consensus.cc:482] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:52.931754 20315 raft_consensus.cc:3037] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.932569 20315 raft_consensus.cc:504] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.932864 20315 leader_election.cc:288] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:52.933207 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1b710faf264200a2394303e4195edf" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:52.933233 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1b710faf264200a2394303e4195edf" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:52.933308 12716 raft_consensus.cc:3037] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.933320 12849 raft_consensus.cc:3037] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:52.934139 12716 raft_consensus.cc:2454] T bd1b710faf264200a2394303e4195edf P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:52.934221 12849 raft_consensus.cc:2454] T bd1b710faf264200a2394303e4195edf P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:52.934430 14891 leader_election.cc:302] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:52.934614 20315 raft_consensus.cc:2781] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:52.934703 20315 raft_consensus.cc:686] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:52.934906 20315 consensus_queue.cc:237] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:52.937054 12443 catalog_manager.cc:5484] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:53.002521 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:53.008270 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 3f01b0e66b1241bf95c09833004ac790 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac]), partition=
I20240501 14:09:53.008366 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 3f01b0e66b1241bf95c09833004ac790 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac]), partition=
I20240501 14:09:53.008575 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f01b0e66b1241bf95c09833004ac790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.008611 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f01b0e66b1241bf95c09833004ac790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.009058 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 3f01b0e66b1241bf95c09833004ac790 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac]), partition=
I20240501 14:09:53.009260 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f01b0e66b1241bf95c09833004ac790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.010759 20630 tablet_bootstrap.cc:492] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:53.010762 20693 tablet_bootstrap.cc:492] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:53.011444 20629 tablet_bootstrap.cc:492] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:53.011691 20693 tablet_bootstrap.cc:654] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.011837 20630 tablet_bootstrap.cc:654] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.012437 20629 tablet_bootstrap.cc:654] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.013150 20630 tablet_bootstrap.cc:492] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:53.013234 20630 ts_tablet_manager.cc:1367] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:53.013516 20629 tablet_bootstrap.cc:492] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:53.013588 20629 ts_tablet_manager.cc:1367] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:53.013713 20630 raft_consensus.cc:348] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.013810 20630 raft_consensus.cc:374] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.013841 20630 raft_consensus.cc:729] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.013964 20630 consensus_queue.cc:260] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.014025 20629 raft_consensus.cc:348] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.014132 20629 raft_consensus.cc:374] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.014163 20629 raft_consensus.cc:729] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.014267 20629 consensus_queue.cc:260] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.014520 20629 ts_tablet_manager.cc:1398] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.016083 20693 tablet_bootstrap.cc:492] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:53.016160 20630 ts_tablet_manager.cc:1398] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:53.016219 20693 ts_tablet_manager.cc:1367] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:53.016724 20693 raft_consensus.cc:348] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.016871 20693 raft_consensus.cc:374] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.016906 20693 raft_consensus.cc:729] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.017204 20693 consensus_queue.cc:260] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.017549 20693 ts_tablet_manager.cc:1398] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.086484 20622 raft_consensus.cc:482] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.086589 20622 raft_consensus.cc:504] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.086956 20622 leader_election.cc:288] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:53.087311 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01b0e66b1241bf95c09833004ac790" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:53.087311 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01b0e66b1241bf95c09833004ac790" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:53.087447 12849 raft_consensus.cc:2454] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:53.087447 14958 raft_consensus.cc:2454] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:53.087749 12916 leader_election.cc:302] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:53.087932 20622 raft_consensus.cc:2781] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.087996 20622 raft_consensus.cc:482] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.088042 20622 raft_consensus.cc:3037] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.089103 20622 raft_consensus.cc:504] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.089403 20622 leader_election.cc:288] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:53.089684 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01b0e66b1241bf95c09833004ac790" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:53.089821 14958 raft_consensus.cc:3037] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.089812 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01b0e66b1241bf95c09833004ac790" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:53.089928 12849 raft_consensus.cc:3037] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.090641 14958 raft_consensus.cc:2454] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:53.090775 12849 raft_consensus.cc:2454] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:53.090945 12914 leader_election.cc:302] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:53.091094 20622 raft_consensus.cc:2781] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.091198 20622 raft_consensus.cc:686] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:53.091331 20622 consensus_queue.cc:237] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:53.093640 12443 catalog_manager.cc:5484] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:53.209093 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:53.213479 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:53.220487 14958 raft_consensus.cc:1261] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:53.220786 12849 raft_consensus.cc:1261] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:53.221180 20311 consensus_queue.cc:1035] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.221365 20622 consensus_queue.cc:1035] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.224596 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 3f01b0e66b1241bf95c09833004ac790 (table rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac]). Expected version 1 got 0
I20240501 14:09:53.224598 20712 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.225103 20712 tablet.cc:1620] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:53.225126 12438 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 3f01b0e66b1241bf95c09833004ac790 (table rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac]). Expected version 1 got 0
I20240501 14:09:53.228144 20447 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.228452 20447 tablet.cc:1620] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:53.230155 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac] alter complete (version 1)
I20240501 14:09:53.244374 20651 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:53.244716 20651 tablet.cc:1620] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:53.283576 20538 consensus_queue.cc:1035] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.288362 20276 consensus_queue.cc:1035] T 92603ceaeb2d4e279507d27a579dc60f P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:53.289834 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:53.333787 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:53.338661 20712 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.339041 20712 tablet.cc:1620] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:53.339115 20651 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.339438 20651 tablet.cc:1620] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:53.340106 20447 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:53.340404 20447 tablet.cc:1620] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:53.344658 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=dc7e5969663143338dff0e6f098818ac] alter complete (version 2)
W20240501 14:09:53.361711 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:53.422539 20152 consensus_queue.cc:1035] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.425833 20330 consensus_queue.cc:1035] T 1e5c69ea0a594bcfb775f1fb884a286a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.435199 20451 consensus_queue.cc:1035] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.439982 20315 consensus_queue.cc:1035] T bd1b710faf264200a2394303e4195edf P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.446938 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:53.451329 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:53.453311 20712 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.453797 20651 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.454744 20447 tablet.cc:1722] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:53.461822 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1755e563a846269d484fe2db88d96c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.461948 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1755e563a846269d484fe2db88d96c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.462121 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1755e563a846269d484fe2db88d96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.462273 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1755e563a846269d484fe2db88d96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.463104 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 369162a18a6a44dd9f9292960f1aee64 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.463418 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369162a18a6a44dd9f9292960f1aee64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.463362 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 81ac77616f4d474eaa51cbad34fbd1c6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.463627 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ac77616f4d474eaa51cbad34fbd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.464624 20693 tablet_bootstrap.cc:492] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:53.465605 20693 tablet_bootstrap.cc:654] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.465574 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 369162a18a6a44dd9f9292960f1aee64 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.465858 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369162a18a6a44dd9f9292960f1aee64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.466091 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 3b1755e563a846269d484fe2db88d96c (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.466320 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b1755e563a846269d484fe2db88d96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.463104 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 81ac77616f4d474eaa51cbad34fbd1c6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.467281 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ac77616f4d474eaa51cbad34fbd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.467423 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 81ac77616f4d474eaa51cbad34fbd1c6 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.467700 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ac77616f4d474eaa51cbad34fbd1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.468221 20737 tablet_bootstrap.cc:492] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:53.468429 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 369162a18a6a44dd9f9292960f1aee64 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=c47d19f0daa442b994368497695fb28b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:53.468715 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369162a18a6a44dd9f9292960f1aee64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.469465 20737 tablet_bootstrap.cc:654] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.471045 20693 tablet_bootstrap.cc:492] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:53.471144 20693 ts_tablet_manager.cc:1367] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:53.471478 20737 tablet_bootstrap.cc:492] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:53.471566 20737 ts_tablet_manager.cc:1367] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:53.471619 20693 raft_consensus.cc:348] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.471737 20693 raft_consensus.cc:374] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.471774 20693 raft_consensus.cc:729] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.471905 20693 consensus_queue.cc:260] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.472079 20737 raft_consensus.cc:348] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.472179 20737 raft_consensus.cc:374] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.472208 20693 ts_tablet_manager.cc:1398] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.472214 20737 raft_consensus.cc:729] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.472337 20630 tablet_bootstrap.cc:492] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:53.472335 20737 consensus_queue.cc:260] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.472389 20693 tablet_bootstrap.cc:492] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:53.472618 20737 ts_tablet_manager.cc:1398] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.472815 20737 tablet_bootstrap.cc:492] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:53.473178 20735 tablet_bootstrap.cc:492] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:53.473282 20693 tablet_bootstrap.cc:654] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.473348 20630 tablet_bootstrap.cc:654] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.473778 20737 tablet_bootstrap.cc:654] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.474167 20734 tablet_bootstrap.cc:492] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:53.474313 20735 tablet_bootstrap.cc:654] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.475287 20734 tablet_bootstrap.cc:654] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.475648 20738 tablet_bootstrap.cc:492] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:53.476270 20630 tablet_bootstrap.cc:492] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:53.476373 20630 ts_tablet_manager.cc:1367] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:53.476775 20738 tablet_bootstrap.cc:654] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.476821 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=dc7e5969663143338dff0e6f098818ac] alter complete (version 3)
I20240501 14:09:53.476833 20630 raft_consensus.cc:348] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.476948 20630 raft_consensus.cc:374] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.476984 20630 raft_consensus.cc:729] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.477108 20630 consensus_queue.cc:260] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.479213 20735 tablet_bootstrap.cc:492] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:53.479314 20630 ts_tablet_manager.cc:1398] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:53.479801 20735 ts_tablet_manager.cc:1367] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:53.480370 20735 raft_consensus.cc:348] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.480935 20738 tablet_bootstrap.cc:492] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:53.481050 20735 raft_consensus.cc:374] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.481109 20735 raft_consensus.cc:729] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.481264 20735 consensus_queue.cc:260] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.481566 20735 ts_tablet_manager.cc:1398] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:53.481757 20735 tablet_bootstrap.cc:492] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:53.482393 20738 ts_tablet_manager.cc:1367] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:53.482987 20738 raft_consensus.cc:348] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.483356 20738 raft_consensus.cc:374] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.483485 20738 raft_consensus.cc:729] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.484907 20738 consensus_queue.cc:260] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.485590 20738 ts_tablet_manager.cc:1398] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:53.485816 20735 tablet_bootstrap.cc:654] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
W20240501 14:09:53.486500 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:53.487511 20693 tablet_bootstrap.cc:492] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:53.487625 20693 ts_tablet_manager.cc:1367] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:09:53.487691 20735 tablet_bootstrap.cc:492] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:53.487771 20735 ts_tablet_manager.cc:1367] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:53.488111 20693 raft_consensus.cc:348] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.488163 20735 raft_consensus.cc:348] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.488220 20693 raft_consensus.cc:374] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.488252 20693 raft_consensus.cc:729] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.488257 20735 raft_consensus.cc:374] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.488292 20735 raft_consensus.cc:729] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.488379 20693 consensus_queue.cc:260] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.488416 20735 consensus_queue.cc:260] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.488675 20693 ts_tablet_manager.cc:1398] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.488742 20735 ts_tablet_manager.cc:1398] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.491951 20734 tablet_bootstrap.cc:492] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:53.492153 20734 ts_tablet_manager.cc:1367] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 14:09:53.492489 20737 tablet_bootstrap.cc:492] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:53.492596 20737 ts_tablet_manager.cc:1367] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 14:09:53.492758 20734 raft_consensus.cc:348] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.492872 20734 raft_consensus.cc:374] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.492908 20734 raft_consensus.cc:729] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.493088 20734 consensus_queue.cc:260] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.493103 20737 raft_consensus.cc:348] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.493206 20737 raft_consensus.cc:374] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.493245 20737 raft_consensus.cc:729] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.493373 20737 consensus_queue.cc:260] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.493534 20734 ts_tablet_manager.cc:1398] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.493703 20737 ts_tablet_manager.cc:1398] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 14:09:53.504282 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:53.518493 20713 raft_consensus.cc:482] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.518622 20713 raft_consensus.cc:504] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.519141 20713 leader_election.cc:288] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:53.519420 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1755e563a846269d484fe2db88d96c" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:53.519567 12716 raft_consensus.cc:2454] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:53.519523 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1755e563a846269d484fe2db88d96c" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:53.519661 15222 raft_consensus.cc:2454] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:53.519824 12913 leader_election.cc:302] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:53.520028 20713 raft_consensus.cc:2781] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.520087 20713 raft_consensus.cc:482] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.520112 20713 raft_consensus.cc:3037] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.521036 20713 raft_consensus.cc:504] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.521404 20713 leader_election.cc:288] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:53.521687 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1755e563a846269d484fe2db88d96c" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:53.521699 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b1755e563a846269d484fe2db88d96c" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:53.521804 12716 raft_consensus.cc:3037] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.521816 15222 raft_consensus.cc:3037] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.522661 12716 raft_consensus.cc:2454] T 3b1755e563a846269d484fe2db88d96c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:53.522778 15222 raft_consensus.cc:2454] T 3b1755e563a846269d484fe2db88d96c P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:53.522985 12913 leader_election.cc:302] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:53.523171 20713 raft_consensus.cc:2781] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.523255 20713 raft_consensus.cc:686] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:53.523372 20713 consensus_queue.cc:237] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.525727 12443 catalog_manager.cc:5484] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:53.580998 20659 raft_consensus.cc:482] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.581131 20659 raft_consensus.cc:504] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.581668 20659 leader_election.cc:288] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:53.582036 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369162a18a6a44dd9f9292960f1aee64" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:53.582182 12585 raft_consensus.cc:2454] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:53.582422 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369162a18a6a44dd9f9292960f1aee64" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:53.582541 15222 raft_consensus.cc:2454] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:53.582513 12784 leader_election.cc:302] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:53.582698 20659 raft_consensus.cc:2781] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.582775 20659 raft_consensus.cc:482] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.582818 20659 raft_consensus.cc:3037] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.583845 20659 raft_consensus.cc:504] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.584152 20659 leader_election.cc:288] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:53.584456 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369162a18a6a44dd9f9292960f1aee64" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:53.584573 12585 raft_consensus.cc:3037] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.585556 12585 raft_consensus.cc:2454] T 369162a18a6a44dd9f9292960f1aee64 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:53.585778 15222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369162a18a6a44dd9f9292960f1aee64" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:53.585826 12784 leader_election.cc:302] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:53.585875 15222 raft_consensus.cc:3037] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.585979 20659 raft_consensus.cc:2781] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.586050 20659 raft_consensus.cc:686] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:53.586191 20659 consensus_queue.cc:237] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:53.586951 15222 raft_consensus.cc:2454] T 369162a18a6a44dd9f9292960f1aee64 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:53.589437 12443 catalog_manager.cc:5484] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:53.591109 12440 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:53.591245 12440 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:53.593151 12440 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 3f01b0e66b1241bf95c09833004ac790
I20240501 14:09:53.593737 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01b0e66b1241bf95c09833004ac790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:53 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:53.594026 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01b0e66b1241bf95c09833004ac790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:53 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:53.594050 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01b0e66b1241bf95c09833004ac790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:53 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:53.594261 20760 tablet_replica.cc:330] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:53.594439 20760 raft_consensus.cc:2229] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:53.594584 20761 tablet_replica.cc:330] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:53.594687 20762 tablet_replica.cc:330] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:53.594702 20760 raft_consensus.cc:2258] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.595167 20761 raft_consensus.cc:2229] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.594857 20762 raft_consensus.cc:2229] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:53.595350 20761 raft_consensus.cc:2258] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.595357 20760 ts_tablet_manager.cc:1875] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:53.595467 20762 raft_consensus.cc:2258] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:53.595988 20761 ts_tablet_manager.cc:1875] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:53.597880 20760 ts_tablet_manager.cc:1888] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:53.597981 20760 log.cc:1196] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/3f01b0e66b1241bf95c09833004ac790
I20240501 14:09:53.598130 20761 ts_tablet_manager.cc:1888] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:53.598229 20761 log.cc:1196] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f01b0e66b1241bf95c09833004ac790
I20240501 14:09:53.598336 20760 ts_tablet_manager.cc:1909] T 3f01b0e66b1241bf95c09833004ac790 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:53.598603 20761 ts_tablet_manager.cc:1909] T 3f01b0e66b1241bf95c09833004ac790 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:53.599103 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 3f01b0e66b1241bf95c09833004ac790 (table rebalancer_dynamic_table_new_name [id=dc7e5969663143338dff0e6f098818ac]) successfully deleted
I20240501 14:09:53.599857 20762 ts_tablet_manager.cc:1875] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:53.601933 20762 ts_tablet_manager.cc:1888] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:53.602006 20762 log.cc:1196] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/3f01b0e66b1241bf95c09833004ac790
I20240501 14:09:53.602236 20762 ts_tablet_manager.cc:1909] T 3f01b0e66b1241bf95c09833004ac790 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:53.602849 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 3f01b0e66b1241bf95c09833004ac790 (table rebalancer_dynamic_table_new_name [id=dc7e5969663143338dff0e6f098818ac]) successfully deleted
I20240501 14:09:53.602941 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 3f01b0e66b1241bf95c09833004ac790 (table rebalancer_dynamic_table_new_name [id=dc7e5969663143338dff0e6f098818ac]) successfully deleted
W20240501 14:09:53.655766 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:53.694662 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:53.700471 12698 tablet_service.cc:1467] Processing CreateTablet for tablet bdb6575a819f439f9850672d93f67f41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]), partition=
I20240501 14:09:53.700491 15331 tablet_service.cc:1467] Processing CreateTablet for tablet bdb6575a819f439f9850672d93f67f41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]), partition=
I20240501 14:09:53.700600 12566 tablet_service.cc:1467] Processing CreateTablet for tablet bdb6575a819f439f9850672d93f67f41 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]), partition=
I20240501 14:09:53.700778 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb6575a819f439f9850672d93f67f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.700780 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb6575a819f439f9850672d93f67f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.700810 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb6575a819f439f9850672d93f67f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:53.702672 20734 tablet_bootstrap.cc:492] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:53.703060 20735 tablet_bootstrap.cc:492] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:53.703589 20734 tablet_bootstrap.cc:654] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.703969 20735 tablet_bootstrap.cc:654] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.704082 20738 tablet_bootstrap.cc:492] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:53.705008 20738 tablet_bootstrap.cc:654] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:53.705363 20734 tablet_bootstrap.cc:492] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:53.705363 20735 tablet_bootstrap.cc:492] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:53.705456 20735 ts_tablet_manager.cc:1367] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:53.705454 20734 ts_tablet_manager.cc:1367] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:53.705891 20735 raft_consensus.cc:348] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.705891 20734 raft_consensus.cc:348] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.706017 20735 raft_consensus.cc:374] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.706028 20734 raft_consensus.cc:374] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.706059 20735 raft_consensus.cc:729] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.706063 20734 raft_consensus.cc:729] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.706189 20734 consensus_queue.cc:260] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.706189 20735 consensus_queue.cc:260] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.706455 20735 ts_tablet_manager.cc:1398] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.706455 20734 ts_tablet_manager.cc:1398] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:53.707021 20738 tablet_bootstrap.cc:492] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:53.707116 20738 ts_tablet_manager.cc:1367] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:53.707561 20738 raft_consensus.cc:348] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.707675 20738 raft_consensus.cc:374] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:53.707710 20738 raft_consensus.cc:729] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:53.707870 20738 consensus_queue.cc:260] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.708148 20738 ts_tablet_manager.cc:1398] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:53.789319 20713 raft_consensus.cc:482] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.789449 20713 raft_consensus.cc:504] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.789739 20538 raft_consensus.cc:482] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:53.789883 20538 raft_consensus.cc:504] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.790040 20713 leader_election.cc:288] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:53.790231 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:53.790341 15351 raft_consensus.cc:2454] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:53.790405 20538 leader_election.cc:288] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:53.790519 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:53.790692 12585 raft_consensus.cc:2454] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:53.790957 12915 leader_election.cc:302] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:53.790959 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:53.791090 12584 raft_consensus.cc:2454] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:53.791119 20713 raft_consensus.cc:2781] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.791157 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:53.791162 20713 raft_consensus.cc:482] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.791283 20713 raft_consensus.cc:3037] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.791430 15288 leader_election.cc:302] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:53.791603 20538 raft_consensus.cc:2781] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:53.791666 20538 raft_consensus.cc:482] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:53.791704 20538 raft_consensus.cc:3037] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.792580 20713 raft_consensus.cc:504] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.792762 12978 raft_consensus.cc:2379] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f23f426391724d6688aafffe7ee017d8 in current term 1: Already voted for candidate a6e94562bee149c2ace25672f0f9bf50 in this term.
I20240501 14:09:53.792775 20538 raft_consensus.cc:504] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.793108 20538 leader_election.cc:288] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:53.793349 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:53.793471 12978 raft_consensus.cc:2379] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f23f426391724d6688aafffe7ee017d8 in current term 1: Already voted for candidate a6e94562bee149c2ace25672f0f9bf50 in this term.
I20240501 14:09:53.793578 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:53.793613 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:53.793684 12585 raft_consensus.cc:3037] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:53.793963 20713 leader_election.cc:288] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:53.794624 12585 raft_consensus.cc:2454] T 81ac77616f4d474eaa51cbad34fbd1c6 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:53.795100 15288 leader_election.cc:302] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:53.795259 20276 raft_consensus.cc:2781] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:53.795341 20276 raft_consensus.cc:686] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:53.795475 20276 consensus_queue.cc:237] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:53.796407 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ac77616f4d474eaa51cbad34fbd1c6" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:53.796885 12916 leader_election.cc:302] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50; no voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8
I20240501 14:09:53.797055 20713 raft_consensus.cc:2726] T 81ac77616f4d474eaa51cbad34fbd1c6 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:09:53.798207 12440 catalog_manager.cc:5484] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:53.833531 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:53.925364 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:53.974467 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:53.989439 20361 consensus_queue.cc:1035] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:53.992545 20659 consensus_queue.cc:1035] T 369162a18a6a44dd9f9292960f1aee64 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:54.019143 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:09:54.040686 20538 raft_consensus.cc:482] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.040802 20538 raft_consensus.cc:504] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.041152 20538 leader_election.cc:288] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:54.041576 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb6575a819f439f9850672d93f67f41" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:54.041721 12717 raft_consensus.cc:2454] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:54.041729 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb6575a819f439f9850672d93f67f41" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:54.041858 12584 raft_consensus.cc:2454] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:54.042068 15286 leader_election.cc:302] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:54.042227 20538 raft_consensus.cc:2781] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.042302 20538 raft_consensus.cc:482] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.042340 20538 raft_consensus.cc:3037] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.043406 20538 raft_consensus.cc:504] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.043743 20538 leader_election.cc:288] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:54.044102 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb6575a819f439f9850672d93f67f41" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:54.044251 12717 raft_consensus.cc:3037] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.044217 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb6575a819f439f9850672d93f67f41" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:54.044332 12584 raft_consensus.cc:3037] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.045279 12717 raft_consensus.cc:2454] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:54.045363 12584 raft_consensus.cc:2454] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:54.045625 15286 leader_election.cc:302] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:54.045868 20538 raft_consensus.cc:2781] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.045969 20538 raft_consensus.cc:686] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:54.046105 20538 consensus_queue.cc:237] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.048789 12443 catalog_manager.cc:5484] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:54.059765 20713 consensus_queue.cc:1035] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.066977 20798 consensus_queue.cc:1035] T 3b1755e563a846269d484fe2db88d96c P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 14:09:54.109758 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:54.171240 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:54.175719 12584 raft_consensus.cc:1261] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:54.175937 12717 raft_consensus.cc:1261] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:54.176225 20538 consensus_queue.cc:1035] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.176384 20276 consensus_queue.cc:1035] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.179363 12440 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]). Expected version 1 got 0
I20240501 14:09:54.179315 20814 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.179677 20814 tablet.cc:1620] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:54.180191 12443 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]). Expected version 1 got 0
I20240501 14:09:54.181695 20811 mvcc.cc:205] Tried to move back new op lower bound from 7022889345740263424 to 7022889345214431232. Current Snapshot: MvccSnapshot[applied={T|T < 7022889345740263424}]
I20240501 14:09:54.182219 12440 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]). Expected version 1 got 0
I20240501 14:09:54.182932 20815 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.183235 20815 tablet.cc:1620] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:54.184554 12438 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]). Expected version 1 got 0
I20240501 14:09:54.186682 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6] alter complete (version 1)
I20240501 14:09:54.187883 20816 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.188243 20816 tablet.cc:1620] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:54.189219 12438 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6]). Expected version 1 got 0
I20240501 14:09:54.290462 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:54.294147 20538 consensus_queue.cc:1035] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.297328 20815 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.297874 20815 tablet.cc:1620] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:54.298090 20276 consensus_queue.cc:1035] T 81ac77616f4d474eaa51cbad34fbd1c6 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.298272 20814 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.298558 20814 tablet.cc:1620] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:09:54.299573 20816 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.299896 20816 tablet.cc:1620] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:54.307418 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=cec6d4c704b64a89a62088aa2cbccbc6] alter complete (version 2)
W20240501 14:09:54.339354 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:54.343592 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:54.350155 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7daa09da3c45ebb550dcc5362446d2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.350296 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7daa09da3c45ebb550dcc5362446d2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.350481 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7daa09da3c45ebb550dcc5362446d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.350519 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7daa09da3c45ebb550dcc5362446d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.351023 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 251f09259f074578a00d57721f4bd34b (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.351114 12566 tablet_service.cc:1467] Processing CreateTablet for tablet b5a710212765496eb8b6969326d1e6d8 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.351303 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251f09259f074578a00d57721f4bd34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.351352 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a710212765496eb8b6969326d1e6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.353444 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 251f09259f074578a00d57721f4bd34b (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.353564 12827 tablet_service.cc:1467] Processing CreateTablet for tablet b5a710212765496eb8b6969326d1e6d8 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.353785 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a710212765496eb8b6969326d1e6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.354174 12698 tablet_service.cc:1467] Processing CreateTablet for tablet b5a710212765496eb8b6969326d1e6d8 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.354231 20822 tablet_bootstrap.cc:492] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:54.354259 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 251f09259f074578a00d57721f4bd34b (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.354451 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a710212765496eb8b6969326d1e6d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.354506 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251f09259f074578a00d57721f4bd34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.355319 20823 tablet_bootstrap.cc:492] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:54.356312 20823 tablet_bootstrap.cc:654] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.356834 20825 tablet_bootstrap.cc:492] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:54.357203 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251f09259f074578a00d57721f4bd34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.357314 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7daa09da3c45ebb550dcc5362446d2 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=fe2d2433ce7342428665daf74229661a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:54.357578 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7daa09da3c45ebb550dcc5362446d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.357707 20825 tablet_bootstrap.cc:654] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.359869 20827 tablet_bootstrap.cc:492] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:54.360045 20822 tablet_bootstrap.cc:654] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.360800 20821 tablet_bootstrap.cc:492] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:54.360977 20827 tablet_bootstrap.cc:654] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.361449 20823 tablet_bootstrap.cc:492] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:54.361534 20823 ts_tablet_manager.cc:1367] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:54.361828 20821 tablet_bootstrap.cc:654] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.362032 20827 tablet_bootstrap.cc:492] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:54.362120 20827 ts_tablet_manager.cc:1367] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:54.362154 20823 raft_consensus.cc:348] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.362267 20823 raft_consensus.cc:374] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.362298 20823 raft_consensus.cc:729] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.362458 20823 consensus_queue.cc:260] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.362496 20820 tablet_bootstrap.cc:492] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:54.362572 20827 raft_consensus.cc:348] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.362671 20827 raft_consensus.cc:374] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.362704 20827 raft_consensus.cc:729] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.362874 20823 ts_tablet_manager.cc:1398] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:09:54.362839 20827 consensus_queue.cc:260] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.363143 20827 ts_tablet_manager.cc:1398] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.363556 20820 tablet_bootstrap.cc:654] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.364890 20825 tablet_bootstrap.cc:492] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:54.364984 20825 ts_tablet_manager.cc:1367] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:54.365465 20825 raft_consensus.cc:348] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.365597 20825 raft_consensus.cc:374] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.365634 20825 raft_consensus.cc:729] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.365779 20825 consensus_queue.cc:260] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.366067 20821 tablet_bootstrap.cc:492] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:54.366120 20825 ts_tablet_manager.cc:1398] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.366151 20821 ts_tablet_manager.cc:1367] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:54.366601 20821 raft_consensus.cc:348] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.366691 20821 raft_consensus.cc:374] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.366719 20821 raft_consensus.cc:729] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.366904 20821 consensus_queue.cc:260] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.367218 20821 ts_tablet_manager.cc:1398] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:54.367647 20821 tablet_bootstrap.cc:492] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:54.368597 20821 tablet_bootstrap.cc:654] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.372854 20821 tablet_bootstrap.cc:492] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:54.373019 20821 ts_tablet_manager.cc:1367] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:09:54.374075 20821 raft_consensus.cc:348] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.374322 20821 raft_consensus.cc:374] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.374401 20821 raft_consensus.cc:729] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.374590 20821 consensus_queue.cc:260] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.375126 20821 ts_tablet_manager.cc:1398] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:54.379109 20824 tablet_bootstrap.cc:492] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:54.380452 20824 tablet_bootstrap.cc:654] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.381939 20822 tablet_bootstrap.cc:492] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:54.382052 20822 ts_tablet_manager.cc:1367] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.002s
I20240501 14:09:54.382444 20822 raft_consensus.cc:348] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.382542 20822 raft_consensus.cc:374] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.382582 20822 raft_consensus.cc:729] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.382721 20822 consensus_queue.cc:260] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.383282 20822 ts_tablet_manager.cc:1398] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:54.383816 20824 tablet_bootstrap.cc:492] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:54.383894 20824 ts_tablet_manager.cc:1367] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:54.384415 20824 raft_consensus.cc:348] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.384557 20824 raft_consensus.cc:374] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.384594 20824 raft_consensus.cc:729] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.384919 20824 consensus_queue.cc:260] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.385229 20824 ts_tablet_manager.cc:1398] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.386121 20820 tablet_bootstrap.cc:492] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:54.386214 20820 ts_tablet_manager.cc:1367] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 14:09:54.386649 20820 raft_consensus.cc:348] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.386758 20820 raft_consensus.cc:374] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.386785 20820 raft_consensus.cc:729] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.386894 20820 consensus_queue.cc:260] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.387141 20820 ts_tablet_manager.cc:1398] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.387290 20820 tablet_bootstrap.cc:492] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:54.388252 20820 tablet_bootstrap.cc:654] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.391287 20820 tablet_bootstrap.cc:492] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:54.391420 20820 ts_tablet_manager.cc:1367] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:54.391851 20820 raft_consensus.cc:348] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.392004 20820 raft_consensus.cc:374] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.392081 20820 raft_consensus.cc:729] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.393513 20820 consensus_queue.cc:260] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.393886 20820 ts_tablet_manager.cc:1398] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:54.410851 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:54.417954 20814 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:54.418816 20816 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:54.420642 20815 tablet.cc:1722] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:54.427708 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=cec6d4c704b64a89a62088aa2cbccbc6] alter complete (version 3)
I20240501 14:09:54.433312 20796 raft_consensus.cc:482] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.433421 20796 raft_consensus.cc:504] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.433810 20796 leader_election.cc:288] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:54.434162 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251f09259f074578a00d57721f4bd34b" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:54.434322 15351 raft_consensus.cc:2454] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:54.434346 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251f09259f074578a00d57721f4bd34b" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:54.434463 14957 raft_consensus.cc:2454] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:54.434702 12916 leader_election.cc:302] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:54.434949 20796 raft_consensus.cc:2781] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.434998 20796 raft_consensus.cc:482] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.435024 20796 raft_consensus.cc:3037] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.436169 20796 raft_consensus.cc:504] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.436492 20796 leader_election.cc:288] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:54.436805 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251f09259f074578a00d57721f4bd34b" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:54.436836 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251f09259f074578a00d57721f4bd34b" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:54.436949 15351 raft_consensus.cc:3037] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.436964 14957 raft_consensus.cc:3037] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.437747 15351 raft_consensus.cc:2454] T 251f09259f074578a00d57721f4bd34b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:54.437803 14957 raft_consensus.cc:2454] T 251f09259f074578a00d57721f4bd34b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:54.438048 12916 leader_election.cc:302] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:54.438246 20796 raft_consensus.cc:2781] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.438338 20796 raft_consensus.cc:686] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:54.438462 20796 consensus_queue.cc:237] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:54.440703 12440 catalog_manager.cc:5484] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:54.443526 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:54.445683 20788 raft_consensus.cc:482] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.445778 20788 raft_consensus.cc:504] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.446205 20788 leader_election.cc:288] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:54.446561 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7daa09da3c45ebb550dcc5362446d2" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:54.446624 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7daa09da3c45ebb550dcc5362446d2" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:54.446681 12848 raft_consensus.cc:2454] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:54.446790 14957 raft_consensus.cc:2454] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:54.447077 15155 leader_election.cc:302] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:54.447243 20788 raft_consensus.cc:2781] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.447309 20788 raft_consensus.cc:482] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.447348 20788 raft_consensus.cc:3037] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.448249 20788 raft_consensus.cc:504] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.448626 20788 leader_election.cc:288] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:54.448860 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7daa09da3c45ebb550dcc5362446d2" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:54.448890 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7daa09da3c45ebb550dcc5362446d2" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:54.449007 14957 raft_consensus.cc:3037] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.449002 12848 raft_consensus.cc:3037] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.450003 12848 raft_consensus.cc:2454] T 8b7daa09da3c45ebb550dcc5362446d2 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:54.450016 14957 raft_consensus.cc:2454] T 8b7daa09da3c45ebb550dcc5362446d2 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:54.450292 15155 leader_election.cc:302] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:54.450439 20788 raft_consensus.cc:2781] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.450542 20788 raft_consensus.cc:686] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:54.450702 20788 consensus_queue.cc:237] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:54.454301 12440 catalog_manager.cc:5484] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:54.532112 12440 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:54.532271 12440 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:54.534322 12440 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet bdb6575a819f439f9850672d93f67f41
I20240501 14:09:54.534963 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb6575a819f439f9850672d93f67f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:54.535009 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb6575a819f439f9850672d93f67f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:54.535187 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet bdb6575a819f439f9850672d93f67f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:54 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:09:54.535321 20850 tablet_replica.cc:330] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:54.535341 20851 tablet_replica.cc:330] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:54.535480 20850 raft_consensus.cc:2229] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.535544 20852 tablet_replica.cc:330] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:09:54.535480 20851 raft_consensus.cc:2229] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:54.535662 20850 raft_consensus.cc:2258] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.535684 20851 raft_consensus.cc:2258] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.535728 20852 raft_consensus.cc:2229] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:54.535977 20852 raft_consensus.cc:2258] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:54.536453 20851 ts_tablet_manager.cc:1875] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.536856 20852 ts_tablet_manager.cc:1875] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.537154 20850 ts_tablet_manager.cc:1875] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:54.539237 20852 ts_tablet_manager.cc:1888] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.539327 20852 log.cc:1196] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/bdb6575a819f439f9850672d93f67f41
I20240501 14:09:54.539443 20850 ts_tablet_manager.cc:1888] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.539525 20850 log.cc:1196] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/bdb6575a819f439f9850672d93f67f41
I20240501 14:09:54.539685 20852 ts_tablet_manager.cc:1909] T bdb6575a819f439f9850672d93f67f41 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:09:54.539860 20850 ts_tablet_manager.cc:1909] T bdb6575a819f439f9850672d93f67f41 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:54.540409 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table_new_name [id=cec6d4c704b64a89a62088aa2cbccbc6]) successfully deleted
I20240501 14:09:54.540524 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table_new_name [id=cec6d4c704b64a89a62088aa2cbccbc6]) successfully deleted
I20240501 14:09:54.540607 20851 ts_tablet_manager.cc:1888] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:54.540695 20851 log.cc:1196] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/bdb6575a819f439f9850672d93f67f41
I20240501 14:09:54.540961 20851 ts_tablet_manager.cc:1909] T bdb6575a819f439f9850672d93f67f41 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:54.541538 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet bdb6575a819f439f9850672d93f67f41 (table rebalancer_dynamic_table_new_name [id=cec6d4c704b64a89a62088aa2cbccbc6]) successfully deleted
W20240501 14:09:54.576776 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:54.592271 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:54.636459 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:54.642545 12827 tablet_service.cc:1467] Processing CreateTablet for tablet f476e8dc76524a6f8e81ec46bc0dfe76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844]), partition=
I20240501 14:09:54.642890 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476e8dc76524a6f8e81ec46bc0dfe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.643260 14937 tablet_service.cc:1467] Processing CreateTablet for tablet f476e8dc76524a6f8e81ec46bc0dfe76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844]), partition=
I20240501 14:09:54.643481 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476e8dc76524a6f8e81ec46bc0dfe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.645035 15202 tablet_service.cc:1467] Processing CreateTablet for tablet f476e8dc76524a6f8e81ec46bc0dfe76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844]), partition=
I20240501 14:09:54.645246 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f476e8dc76524a6f8e81ec46bc0dfe76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:54.645287 20821 tablet_bootstrap.cc:492] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:54.645730 20820 tablet_bootstrap.cc:492] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:54.646394 20821 tablet_bootstrap.cc:654] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.646500 20820 tablet_bootstrap.cc:654] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.647655 20827 tablet_bootstrap.cc:492] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:54.647725 20821 tablet_bootstrap.cc:492] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:54.647796 20821 ts_tablet_manager.cc:1367] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:54.648299 20821 raft_consensus.cc:348] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.648432 20821 raft_consensus.cc:374] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.648466 20821 raft_consensus.cc:729] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.648587 20821 consensus_queue.cc:260] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.648753 20827 tablet_bootstrap.cc:654] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:54.648891 20821 ts_tablet_manager.cc:1398] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:54.649634 20820 tablet_bootstrap.cc:492] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:54.649740 20820 ts_tablet_manager.cc:1367] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:54.650215 20820 raft_consensus.cc:348] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.650343 20820 raft_consensus.cc:374] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.650375 20820 raft_consensus.cc:729] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.650485 20820 consensus_queue.cc:260] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.650820 20820 ts_tablet_manager.cc:1398] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:54.653040 20827 tablet_bootstrap.cc:492] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:54.653121 20827 ts_tablet_manager.cc:1367] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:54.653486 20827 raft_consensus.cc:348] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.653622 20827 raft_consensus.cc:374] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:54.653679 20827 raft_consensus.cc:729] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:54.653816 20827 consensus_queue.cc:260] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.654089 20827 ts_tablet_manager.cc:1398] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:54.663057 20849 raft_consensus.cc:482] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.663179 20849 raft_consensus.cc:504] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.663625 20849 leader_election.cc:288] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:54.663951 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a710212765496eb8b6969326d1e6d8" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:54.664005 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a710212765496eb8b6969326d1e6d8" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:54.664116 12585 raft_consensus.cc:2454] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:54.664117 12717 raft_consensus.cc:2454] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:54.664363 12782 leader_election.cc:302] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:54.664513 20849 raft_consensus.cc:2781] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.664574 20849 raft_consensus.cc:482] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.664600 20849 raft_consensus.cc:3037] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.665741 20849 raft_consensus.cc:504] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.666090 20849 leader_election.cc:288] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:54.666397 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a710212765496eb8b6969326d1e6d8" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:54.666517 12717 raft_consensus.cc:3037] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.666616 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a710212765496eb8b6969326d1e6d8" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:54.666751 12585 raft_consensus.cc:3037] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.667439 12717 raft_consensus.cc:2454] T b5a710212765496eb8b6969326d1e6d8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:54.667703 12782 leader_election.cc:302] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:54.667811 12585 raft_consensus.cc:2454] T b5a710212765496eb8b6969326d1e6d8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:54.667868 20849 raft_consensus.cc:2781] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.668416 20849 raft_consensus.cc:686] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:54.668586 20849 consensus_queue.cc:237] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:54.670758 12443 catalog_manager.cc:5484] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:54.688444 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:54.691967 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:54.720293 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:54.729959 20152 raft_consensus.cc:482] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.730078 20152 raft_consensus.cc:504] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.730485 20152 leader_election.cc:288] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:54.730847 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476e8dc76524a6f8e81ec46bc0dfe76" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:54.730868 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476e8dc76524a6f8e81ec46bc0dfe76" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:54.730995 14957 raft_consensus.cc:2454] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:54.731026 12848 raft_consensus.cc:2454] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:54.731316 15155 leader_election.cc:302] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:54.731480 20152 raft_consensus.cc:2781] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:54.731550 20152 raft_consensus.cc:482] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:54.731606 20152 raft_consensus.cc:3037] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.732648 20152 raft_consensus.cc:504] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.733282 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476e8dc76524a6f8e81ec46bc0dfe76" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:54.733280 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f476e8dc76524a6f8e81ec46bc0dfe76" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:54.733389 14957 raft_consensus.cc:3037] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.733386 12849 raft_consensus.cc:3037] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:54.733387 20152 leader_election.cc:288] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:54.734196 12849 raft_consensus.cc:2454] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:54.734241 14957 raft_consensus.cc:2454] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:54.734469 15157 leader_election.cc:302] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:54.734710 20152 raft_consensus.cc:2781] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:54.734809 20152 raft_consensus.cc:686] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:54.734951 20152 consensus_queue.cc:237] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:54.737449 12443 catalog_manager.cc:5484] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:54.808530 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:54.824113 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:54.827975 20152 consensus_queue.cc:1035] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.828783 20798 consensus_queue.cc:1035] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.835162 20798 consensus_queue.cc:1035] T 251f09259f074578a00d57721f4bd34b P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.835449 20788 consensus_queue.cc:1035] T 8b7daa09da3c45ebb550dcc5362446d2 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:54.840103 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:09:54.846060 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:54.847280 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:54.851162 14958 raft_consensus.cc:1261] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:54.851300 12849 raft_consensus.cc:1261] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:54.851646 20152 consensus_queue.cc:1035] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.851806 20788 consensus_queue.cc:1035] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:54.855491 20889 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.855758 12440 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet f476e8dc76524a6f8e81ec46bc0dfe76 (table rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844]). Expected version 1 got 0
I20240501 14:09:54.855926 20889 tablet.cc:1620] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:54.856781 20890 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.861138 12443 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet f476e8dc76524a6f8e81ec46bc0dfe76 (table rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844]). Expected version 1 got 0
I20240501 14:09:54.861577 20891 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:54.861855 20891 tablet.cc:1620] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:54.863689 20890 tablet.cc:1620] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:54.864233 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844] alter complete (version 1)
W20240501 14:09:54.877334 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:54.967797 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:54.973117 20889 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.973486 20889 tablet.cc:1620] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:54.973773 20890 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.974092 20890 tablet.cc:1620] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:54.977283 20891 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:54.977774 20891 tablet.cc:1620] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:54.980363 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=5379942143974caba5ad9e8ffcf97844] alter complete (version 2)
W20240501 14:09:55.013967 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:55.024629 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:55.050159 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.053813 20361 consensus_queue.cc:1035] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.057075 20849 consensus_queue.cc:1035] T b5a710212765496eb8b6969326d1e6d8 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:55.064692 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.084095 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:55.089501 20889 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.089902 20890 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.094071 20891 tablet.cc:1722] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.097905 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=5379942143974caba5ad9e8ffcf97844] alter complete (version 3)
W20240501 14:09:55.105235 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.199498 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:55.199648 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:55.201807 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet f476e8dc76524a6f8e81ec46bc0dfe76
I20240501 14:09:55.202556 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet f476e8dc76524a6f8e81ec46bc0dfe76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:55.202952 20910 tablet_replica.cc:330] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:55.202977 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet f476e8dc76524a6f8e81ec46bc0dfe76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:55.203042 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet f476e8dc76524a6f8e81ec46bc0dfe76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:55.203137 20910 raft_consensus.cc:2229] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.203276 20910 raft_consensus.cc:2258] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.203342 20911 tablet_replica.cc:330] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:55.203377 20912 tablet_replica.cc:330] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:55.203481 20911 raft_consensus.cc:2229] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:55.203776 20911 raft_consensus.cc:2258] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.203543 20912 raft_consensus.cc:2229] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.204058 20910 ts_tablet_manager.cc:1875] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.204062 20912 raft_consensus.cc:2258] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.204632 20912 ts_tablet_manager.cc:1875] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.204762 20911 ts_tablet_manager.cc:1875] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:09:55.204985 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.207294 20910 ts_tablet_manager.cc:1888] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.207419 20910 log.cc:1196] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/f476e8dc76524a6f8e81ec46bc0dfe76
I20240501 14:09:55.207755 20910 ts_tablet_manager.cc:1909] T f476e8dc76524a6f8e81ec46bc0dfe76 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:55.208194 20911 ts_tablet_manager.cc:1888] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.208279 20911 log.cc:1196] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/f476e8dc76524a6f8e81ec46bc0dfe76
I20240501 14:09:55.208616 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet f476e8dc76524a6f8e81ec46bc0dfe76 (table rebalancer_dynamic_table_new_name [id=5379942143974caba5ad9e8ffcf97844]) successfully deleted
I20240501 14:09:55.208703 20911 ts_tablet_manager.cc:1909] T f476e8dc76524a6f8e81ec46bc0dfe76 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:55.209851 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet f476e8dc76524a6f8e81ec46bc0dfe76 (table rebalancer_dynamic_table_new_name [id=5379942143974caba5ad9e8ffcf97844]) successfully deleted
I20240501 14:09:55.211139 20912 ts_tablet_manager.cc:1888] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.211205 20912 log.cc:1196] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/f476e8dc76524a6f8e81ec46bc0dfe76
I20240501 14:09:55.211447 20912 ts_tablet_manager.cc:1909] T f476e8dc76524a6f8e81ec46bc0dfe76 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:55.212124 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet f476e8dc76524a6f8e81ec46bc0dfe76 (table rebalancer_dynamic_table_new_name [id=5379942143974caba5ad9e8ffcf97844]) successfully deleted
W20240501 14:09:55.214278 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:55.236810 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:55.243827 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 622a022a8ee840d884cd735d7f131f5d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.244211 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622a022a8ee840d884cd735d7f131f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.244841 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 622a022a8ee840d884cd735d7f131f5d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.245119 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622a022a8ee840d884cd735d7f131f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.245169 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1076b0b2204f4ee6b7de27f1c4591f23 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.245447 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1076b0b2204f4ee6b7de27f1c4591f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.246810 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd345bb5cf94f0ab411acb98c1c8195 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.247125 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd345bb5cf94f0ab411acb98c1c8195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.247802 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 1076b0b2204f4ee6b7de27f1c4591f23 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.248062 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1076b0b2204f4ee6b7de27f1c4591f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.248787 20914 tablet_bootstrap.cc:492] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:55.249117 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 622a022a8ee840d884cd735d7f131f5d (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.249374 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622a022a8ee840d884cd735d7f131f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.249464 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd345bb5cf94f0ab411acb98c1c8195 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.249722 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd345bb5cf94f0ab411acb98c1c8195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.249374 20913 tablet_bootstrap.cc:492] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:55.250500 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 1076b0b2204f4ee6b7de27f1c4591f23 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.250806 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1076b0b2204f4ee6b7de27f1c4591f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.250893 20915 tablet_bootstrap.cc:492] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:55.252032 20915 tablet_bootstrap.cc:654] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.251914 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd345bb5cf94f0ab411acb98c1c8195 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=f1689a66237444bdb059197dd6fa79ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:55.252230 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd345bb5cf94f0ab411acb98c1c8195. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:09:55.252977 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.254127 20913 tablet_bootstrap.cc:654] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.255358 20918 tablet_bootstrap.cc:492] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:55.255708 20915 tablet_bootstrap.cc:492] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:55.255858 20915 ts_tablet_manager.cc:1367] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:55.256314 20918 tablet_bootstrap.cc:654] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.256879 20915 raft_consensus.cc:348] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.257138 20915 raft_consensus.cc:374] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.257272 20915 raft_consensus.cc:729] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.257467 20913 tablet_bootstrap.cc:492] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:55.257512 20915 consensus_queue.cc:260] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.257625 20913 ts_tablet_manager.cc:1367] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:55.258073 20913 raft_consensus.cc:348] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.258205 20913 raft_consensus.cc:374] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.258244 20913 raft_consensus.cc:729] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.258363 20913 consensus_queue.cc:260] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.258631 20913 ts_tablet_manager.cc:1398] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:55.259155 20915 ts_tablet_manager.cc:1398] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:55.259534 20913 tablet_bootstrap.cc:492] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
W20240501 14:09:55.259797 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.260751 20913 tablet_bootstrap.cc:654] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.261482 20914 tablet_bootstrap.cc:654] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.263042 20916 tablet_bootstrap.cc:492] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:55.263841 20916 tablet_bootstrap.cc:654] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.265512 20914 tablet_bootstrap.cc:492] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:55.265619 20914 ts_tablet_manager.cc:1367] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 14:09:55.265987 20916 tablet_bootstrap.cc:492] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:55.266085 20916 ts_tablet_manager.cc:1367] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:55.266122 20914 raft_consensus.cc:348] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.266243 20914 raft_consensus.cc:374] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.266302 20914 raft_consensus.cc:729] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.266458 20914 consensus_queue.cc:260] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.266716 20914 ts_tablet_manager.cc:1398] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.266676 20916 raft_consensus.cc:348] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.266806 20916 raft_consensus.cc:374] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.266888 20916 raft_consensus.cc:729] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.267048 20916 consensus_queue.cc:260] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.267359 20916 ts_tablet_manager.cc:1398] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.267966 20914 tablet_bootstrap.cc:492] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:55.268494 20916 tablet_bootstrap.cc:492] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:55.268839 20914 tablet_bootstrap.cc:654] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.269423 20916 tablet_bootstrap.cc:654] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.271371 20914 tablet_bootstrap.cc:492] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:55.271469 20914 ts_tablet_manager.cc:1367] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:55.271907 20914 raft_consensus.cc:348] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.272512 20913 tablet_bootstrap.cc:492] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:55.272610 20913 ts_tablet_manager.cc:1367] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:09:55.271996 20914 raft_consensus.cc:374] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.273088 20914 raft_consensus.cc:729] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.273075 20913 raft_consensus.cc:348] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.273183 20913 raft_consensus.cc:374] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.273224 20913 raft_consensus.cc:729] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.273381 20914 consensus_queue.cc:260] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.273763 20914 ts_tablet_manager.cc:1398] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:55.273825 20918 tablet_bootstrap.cc:492] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:55.273926 20918 ts_tablet_manager.cc:1367] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:09:55.274686 20913 consensus_queue.cc:260] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.275146 20913 ts_tablet_manager.cc:1398] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:55.274427 20918 raft_consensus.cc:348] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.275334 20918 raft_consensus.cc:374] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.275368 20918 raft_consensus.cc:729] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.275566 20918 consensus_queue.cc:260] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.275822 20918 ts_tablet_manager.cc:1398] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:55.275998 20918 tablet_bootstrap.cc:492] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:55.276801 20918 tablet_bootstrap.cc:654] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.277853 20916 tablet_bootstrap.cc:492] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:55.277954 20916 ts_tablet_manager.cc:1367] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:55.278445 20916 raft_consensus.cc:348] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.278561 20916 raft_consensus.cc:374] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.278595 20916 raft_consensus.cc:729] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.278626 20918 tablet_bootstrap.cc:492] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:55.278769 20918 ts_tablet_manager.cc:1367] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:55.278785 20916 consensus_queue.cc:260] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.279112 20916 ts_tablet_manager.cc:1398] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.279145 20918 raft_consensus.cc:348] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.279235 20918 raft_consensus.cc:374] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.279289 20918 raft_consensus.cc:729] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.279404 20918 consensus_queue.cc:260] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.279680 20918 ts_tablet_manager.cc:1398] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.298794 20849 raft_consensus.cc:482] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.298913 20849 raft_consensus.cc:504] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.299297 20849 leader_election.cc:288] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:55.299708 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076b0b2204f4ee6b7de27f1c4591f23" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:55.299707 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076b0b2204f4ee6b7de27f1c4591f23" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:55.299849 12978 raft_consensus.cc:2454] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:55.299857 12584 raft_consensus.cc:2454] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:55.300194 12784 leader_election.cc:302] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:55.300449 20849 raft_consensus.cc:2781] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.300501 20849 raft_consensus.cc:482] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.300536 20849 raft_consensus.cc:3037] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.301395 20849 raft_consensus.cc:504] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.301685 20849 leader_election.cc:288] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:55.301977 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076b0b2204f4ee6b7de27f1c4591f23" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:55.302071 12978 raft_consensus.cc:3037] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.302031 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076b0b2204f4ee6b7de27f1c4591f23" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:55.302181 12584 raft_consensus.cc:3037] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.302942 12978 raft_consensus.cc:2454] T 1076b0b2204f4ee6b7de27f1c4591f23 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:55.303053 12584 raft_consensus.cc:2454] T 1076b0b2204f4ee6b7de27f1c4591f23 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:55.303223 12784 leader_election.cc:302] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:55.303423 20849 raft_consensus.cc:2781] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.303519 20849 raft_consensus.cc:686] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:55.303675 20849 consensus_queue.cc:237] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:55.304494 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:55.306222 12440 catalog_manager.cc:5484] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:55.314570 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed34b1e79f347a3847a292e3af9b7ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]), partition=
I20240501 14:09:55.314673 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed34b1e79f347a3847a292e3af9b7ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]), partition=
I20240501 14:09:55.314832 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed34b1e79f347a3847a292e3af9b7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.314914 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed34b1e79f347a3847a292e3af9b7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.314899 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed34b1e79f347a3847a292e3af9b7ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]), partition=
I20240501 14:09:55.315112 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed34b1e79f347a3847a292e3af9b7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.316599 20918 tablet_bootstrap.cc:492] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:55.317129 20915 tablet_bootstrap.cc:492] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:55.317368 20938 tablet_bootstrap.cc:492] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:55.317399 20918 tablet_bootstrap.cc:654] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.317811 20915 tablet_bootstrap.cc:654] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.318316 20938 tablet_bootstrap.cc:654] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.318569 20918 tablet_bootstrap.cc:492] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:55.318660 20918 ts_tablet_manager.cc:1367] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:55.319252 20918 raft_consensus.cc:348] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.319365 20918 raft_consensus.cc:374] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.319412 20918 raft_consensus.cc:729] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.319542 20918 consensus_queue.cc:260] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.319713 20938 tablet_bootstrap.cc:492] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:55.319795 20938 ts_tablet_manager.cc:1367] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:55.319862 20918 ts_tablet_manager.cc:1398] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:55.320273 20938 raft_consensus.cc:348] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.320430 20915 tablet_bootstrap.cc:492] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:55.320523 20915 ts_tablet_manager.cc:1367] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:55.320751 20938 raft_consensus.cc:374] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.320802 20938 raft_consensus.cc:729] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.320904 20938 consensus_queue.cc:260] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.320999 20915 raft_consensus.cc:348] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.321100 20915 raft_consensus.cc:374] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.321134 20915 raft_consensus.cc:729] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.321249 20915 consensus_queue.cc:260] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.321302 20938 ts_tablet_manager.cc:1398] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.321547 20915 ts_tablet_manager.cc:1398] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.335858 19584 raft_consensus.cc:482] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.335991 19584 raft_consensus.cc:504] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.336450 19584 leader_election.cc:288] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:55.336760 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd345bb5cf94f0ab411acb98c1c8195" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:55.336783 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd345bb5cf94f0ab411acb98c1c8195" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:55.336874 15352 raft_consensus.cc:2454] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:55.336900 12978 raft_consensus.cc:2454] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:55.337185 12653 leader_election.cc:302] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:55.337359 19584 raft_consensus.cc:2781] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.337414 19584 raft_consensus.cc:482] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.337440 19584 raft_consensus.cc:3037] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.338326 19584 raft_consensus.cc:504] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.338640 19584 leader_election.cc:288] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:55.338961 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd345bb5cf94f0ab411acb98c1c8195" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:55.339073 12978 raft_consensus.cc:3037] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.339751 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd345bb5cf94f0ab411acb98c1c8195" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:55.339846 15352 raft_consensus.cc:3037] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.340708 15352 raft_consensus.cc:2454] T 4dd345bb5cf94f0ab411acb98c1c8195 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:55.341006 12654 leader_election.cc:302] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:55.341266 19584 raft_consensus.cc:2781] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.341358 19584 raft_consensus.cc:686] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:55.341490 19584 consensus_queue.cc:237] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:55.343523 12978 raft_consensus.cc:2454] T 4dd345bb5cf94f0ab411acb98c1c8195 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:55.344035 12440 catalog_manager.cc:5484] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:55.351083 20798 raft_consensus.cc:482] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.351184 20798 raft_consensus.cc:504] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.351606 20798 leader_election.cc:288] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:55.351912 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed34b1e79f347a3847a292e3af9b7ca" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:55.351976 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed34b1e79f347a3847a292e3af9b7ca" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:55.352051 12849 raft_consensus.cc:2454] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:55.352082 14958 raft_consensus.cc:2454] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:55.352295 12916 leader_election.cc:302] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:55.352574 20798 raft_consensus.cc:2781] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.352622 20798 raft_consensus.cc:482] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.352674 20798 raft_consensus.cc:3037] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.353598 20798 raft_consensus.cc:504] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.353916 20798 leader_election.cc:288] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:55.354197 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed34b1e79f347a3847a292e3af9b7ca" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:55.354307 14958 raft_consensus.cc:3037] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.354414 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed34b1e79f347a3847a292e3af9b7ca" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:55.354525 12849 raft_consensus.cc:3037] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.355363 14958 raft_consensus.cc:2454] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:55.355603 12849 raft_consensus.cc:2454] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:55.355614 12914 leader_election.cc:302] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:55.355751 20798 raft_consensus.cc:2781] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.355831 20798 raft_consensus.cc:686] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:55.355962 20798 consensus_queue.cc:237] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:55.357978 12440 catalog_manager.cc:5484] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:55.470965 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:55.474941 12849 raft_consensus.cc:1261] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:55.474963 14958 raft_consensus.cc:1261] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:55.475427 20798 consensus_queue.cc:1035] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.475580 20858 consensus_queue.cc:1035] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.479034 20890 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.479362 20890 tablet.cc:1620] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:55.479573 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]). Expected version 1 got 0
I20240501 14:09:55.480247 12440 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]). Expected version 1 got 0
I20240501 14:09:55.481308 12440 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]). Expected version 1 got 0
I20240501 14:09:55.482093 20948 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.482414 20948 tablet.cc:1620] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:55.484251 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e] alter complete (version 1)
I20240501 14:09:55.484655 20891 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:55.484946 20891 tablet.cc:1620] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:55.489004 12440 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e]). Expected version 1 got 0
W20240501 14:09:55.503036 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.510394 20226 raft_consensus.cc:482] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.510493 20226 raft_consensus.cc:504] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.510929 20226 leader_election.cc:288] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:55.511287 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622a022a8ee840d884cd735d7f131f5d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:55.511297 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622a022a8ee840d884cd735d7f131f5d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:55.511442 12584 raft_consensus.cc:2454] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:55.511438 15352 raft_consensus.cc:2454] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:55.511725 12653 leader_election.cc:302] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:55.511941 20226 raft_consensus.cc:2781] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.512004 20226 raft_consensus.cc:482] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.512063 20226 raft_consensus.cc:3037] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.512850 20226 raft_consensus.cc:504] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.513166 20226 leader_election.cc:288] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:55.513460 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622a022a8ee840d884cd735d7f131f5d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:55.513489 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622a022a8ee840d884cd735d7f131f5d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:55.513581 15352 raft_consensus.cc:3037] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.513592 12584 raft_consensus.cc:3037] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.514313 15352 raft_consensus.cc:2454] T 622a022a8ee840d884cd735d7f131f5d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:55.514344 12584 raft_consensus.cc:2454] T 622a022a8ee840d884cd735d7f131f5d P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:55.514587 12654 leader_election.cc:302] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:55.514775 20226 raft_consensus.cc:2781] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.514854 20226 raft_consensus.cc:686] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:55.515009 20226 consensus_queue.cc:237] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:55.517488 12440 catalog_manager.cc:5484] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:55.588122 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:55.593986 20948 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.594326 20948 tablet.cc:1620] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:55.594384 20891 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.594683 20891 tablet.cc:1620] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:55.600126 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=6152a2a7858e46aea457e3a57f74741e] alter complete (version 2)
I20240501 14:09:55.600498 20890 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:55.600790 20890 tablet.cc:1620] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
W20240501 14:09:55.654167 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:55.703708 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:55.710021 20948 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.710556 20890 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.714617 20891 tablet.cc:1722] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:55.716495 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=6152a2a7858e46aea457e3a57f74741e] alter complete (version 3)
I20240501 14:09:55.774668 20961 consensus_queue.cc:1035] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.777941 20961 consensus_queue.cc:1035] T 1076b0b2204f4ee6b7de27f1c4591f23 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:55.798882 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.822587 12440 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:55.822798 12440 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:55.824923 12440 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 2ed34b1e79f347a3847a292e3af9b7ca
I20240501 14:09:55.825485 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed34b1e79f347a3847a292e3af9b7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:55.825618 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed34b1e79f347a3847a292e3af9b7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:55.825898 20976 tablet_replica.cc:330] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:55.825872 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed34b1e79f347a3847a292e3af9b7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:55 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:55.826102 20976 raft_consensus.cc:2229] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.826135 20977 tablet_replica.cc:330] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:55.826210 20975 tablet_replica.cc:330] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:55.826259 20977 raft_consensus.cc:2229] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:55.826407 20977 raft_consensus.cc:2258] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.826367 20975 raft_consensus.cc:2229] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:55.826761 20976 raft_consensus.cc:2258] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.826846 20975 raft_consensus.cc:2258] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:55.827517 20975 ts_tablet_manager.cc:1875] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.827677 20977 ts_tablet_manager.cc:1875] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.829879 20977 ts_tablet_manager.cc:1888] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.829879 20975 ts_tablet_manager.cc:1888] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.829950 20975 log.cc:1196] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/2ed34b1e79f347a3847a292e3af9b7ca
I20240501 14:09:55.829960 20976 ts_tablet_manager.cc:1875] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:55.830034 20977 log.cc:1196] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/2ed34b1e79f347a3847a292e3af9b7ca
I20240501 14:09:55.830312 20977 ts_tablet_manager.cc:1909] T 2ed34b1e79f347a3847a292e3af9b7ca P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:55.830328 20975 ts_tablet_manager.cc:1909] T 2ed34b1e79f347a3847a292e3af9b7ca P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:55.831004 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table_new_name [id=6152a2a7858e46aea457e3a57f74741e]) successfully deleted
I20240501 14:09:55.831071 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table_new_name [id=6152a2a7858e46aea457e3a57f74741e]) successfully deleted
I20240501 14:09:55.832580 20976 ts_tablet_manager.cc:1888] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:55.832653 20976 log.cc:1196] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/2ed34b1e79f347a3847a292e3af9b7ca
I20240501 14:09:55.832937 20976 ts_tablet_manager.cc:1909] T 2ed34b1e79f347a3847a292e3af9b7ca P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:55.833567 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 2ed34b1e79f347a3847a292e3af9b7ca (table rebalancer_dynamic_table_new_name [id=6152a2a7858e46aea457e3a57f74741e]) successfully deleted
I20240501 14:09:55.834721 19584 consensus_queue.cc:1035] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.838867 20979 consensus_queue.cc:1035] T 4dd345bb5cf94f0ab411acb98c1c8195 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:55.852854 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:09:55.885701 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:55.926985 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:55.933053 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc1b298254c4a54868e28913abfeb85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]), partition=
I20240501 14:09:55.933362 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc1b298254c4a54868e28913abfeb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.936192 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc1b298254c4a54868e28913abfeb85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]), partition=
I20240501 14:09:55.936475 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc1b298254c4a54868e28913abfeb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.938850 20995 tablet_bootstrap.cc:492] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:55.938997 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc1b298254c4a54868e28913abfeb85 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]), partition=
I20240501 14:09:55.939226 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc1b298254c4a54868e28913abfeb85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:55.939744 20995 tablet_bootstrap.cc:654] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.940630 20995 tablet_bootstrap.cc:492] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:55.940698 20995 ts_tablet_manager.cc:1367] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:55.940840 20997 tablet_bootstrap.cc:492] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:55.942154 20997 tablet_bootstrap.cc:654] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.942713 20995 raft_consensus.cc:348] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.942986 20995 raft_consensus.cc:374] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.943078 20994 tablet_bootstrap.cc:492] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:55.943094 20995 raft_consensus.cc:729] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.943295 20995 consensus_queue.cc:260] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.943646 20995 ts_tablet_manager.cc:1398] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:55.944135 20997 tablet_bootstrap.cc:492] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:55.944219 20997 ts_tablet_manager.cc:1367] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:55.944377 20994 tablet_bootstrap.cc:654] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:55.944660 20997 raft_consensus.cc:348] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.944773 20997 raft_consensus.cc:374] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.944844 20997 raft_consensus.cc:729] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.945004 20997 consensus_queue.cc:260] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.945335 20997 ts_tablet_manager.cc:1398] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:55.945919 20994 tablet_bootstrap.cc:492] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:55.946007 20994 ts_tablet_manager.cc:1367] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:55.946631 20994 raft_consensus.cc:348] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.946780 20994 raft_consensus.cc:374] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:55.946843 20994 raft_consensus.cc:729] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:55.947010 20994 consensus_queue.cc:260] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.947292 20994 ts_tablet_manager.cc:1398] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:55.969980 20979 consensus_queue.cc:1035] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.973757 19584 consensus_queue.cc:1035] T 622a022a8ee840d884cd735d7f131f5d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:55.983775 20777 raft_consensus.cc:482] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:55.983891 20777 raft_consensus.cc:504] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.984270 20777 leader_election.cc:288] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:55.984659 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc1b298254c4a54868e28913abfeb85" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:55.984680 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc1b298254c4a54868e28913abfeb85" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:55.984788 12585 raft_consensus.cc:2454] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:55.984800 15221 raft_consensus.cc:2454] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:55.985071 14893 leader_election.cc:302] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:55.985242 20777 raft_consensus.cc:2781] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:55.985304 20777 raft_consensus.cc:482] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:55.985339 20777 raft_consensus.cc:3037] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.986349 20777 raft_consensus.cc:504] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.986686 20777 leader_election.cc:288] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:55.987056 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc1b298254c4a54868e28913abfeb85" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:55.987063 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc1b298254c4a54868e28913abfeb85" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:55.987180 15221 raft_consensus.cc:3037] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.987179 12585 raft_consensus.cc:3037] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:55.988168 12585 raft_consensus.cc:2454] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:55.988168 15221 raft_consensus.cc:2454] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:55.988471 14891 leader_election.cc:302] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:55.988670 20777 raft_consensus.cc:2781] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:55.988811 20777 raft_consensus.cc:686] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:55.988952 20777 consensus_queue.cc:237] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:55.991449 12438 catalog_manager.cc:5484] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:56.048135 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:56.055137 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 19c8493cd6b3416ab87340a662b4126d (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.055455 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c8493cd6b3416ab87340a662b4126d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.055517 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 0b32701a9b1847c6b81a603332f1e1c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.055784 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b32701a9b1847c6b81a603332f1e1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.057404 20997 tablet_bootstrap.cc:492] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:56.057648 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 19c8493cd6b3416ab87340a662b4126d (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.057811 20994 tablet_bootstrap.cc:492] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:56.057929 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c8493cd6b3416ab87340a662b4126d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.058434 20997 tablet_bootstrap.cc:654] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.058661 20994 tablet_bootstrap.cc:654] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.060103 20994 tablet_bootstrap.cc:492] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:56.060168 20994 ts_tablet_manager.cc:1367] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:09:56.054646 14937 tablet_service.cc:1467] Processing CreateTablet for tablet aab177cf70c34fc5b6a931b175a14e73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.060606 20994 raft_consensus.cc:348] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.060720 20994 raft_consensus.cc:374] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.060748 20994 raft_consensus.cc:729] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.060850 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab177cf70c34fc5b6a931b175a14e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.060853 20994 consensus_queue.cc:260] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.061121 20994 ts_tablet_manager.cc:1398] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:56.063196 21007 tablet_bootstrap.cc:492] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:56.063557 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 0b32701a9b1847c6b81a603332f1e1c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.063820 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b32701a9b1847c6b81a603332f1e1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.064323 21007 tablet_bootstrap.cc:654] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.064879 12959 tablet_service.cc:1467] Processing CreateTablet for tablet aab177cf70c34fc5b6a931b175a14e73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.065155 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab177cf70c34fc5b6a931b175a14e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.065631 15202 tablet_service.cc:1467] Processing CreateTablet for tablet aab177cf70c34fc5b6a931b175a14e73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.065891 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab177cf70c34fc5b6a931b175a14e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.066989 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 19c8493cd6b3416ab87340a662b4126d (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.067209 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c8493cd6b3416ab87340a662b4126d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.068205 21008 tablet_bootstrap.cc:492] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:56.068842 21009 tablet_bootstrap.cc:492] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:56.069142 20995 tablet_bootstrap.cc:492] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:56.070005 20995 tablet_bootstrap.cc:654] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.070796 21008 tablet_bootstrap.cc:654] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.071851 21009 tablet_bootstrap.cc:654] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.072840 20995 tablet_bootstrap.cc:492] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:56.072911 20995 ts_tablet_manager.cc:1367] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:56.073282 20995 raft_consensus.cc:348] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.073366 20995 raft_consensus.cc:374] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.073390 20995 raft_consensus.cc:729] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.073484 20995 consensus_queue.cc:260] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.073736 20995 ts_tablet_manager.cc:1398] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.073904 20995 tablet_bootstrap.cc:492] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:56.074542 21007 tablet_bootstrap.cc:492] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:56.074652 21007 ts_tablet_manager.cc:1367] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:56.074949 20995 tablet_bootstrap.cc:654] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.075368 21007 raft_consensus.cc:348] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.075475 21007 raft_consensus.cc:374] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.075513 21007 raft_consensus.cc:729] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.075620 21007 consensus_queue.cc:260] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.076903 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 0b32701a9b1847c6b81a603332f1e1c0 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=e9eb070a544e4d62bcd8c14bee68ff2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.077209 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b32701a9b1847c6b81a603332f1e1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.077421 21009 tablet_bootstrap.cc:492] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:56.077551 21009 ts_tablet_manager.cc:1367] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:56.078008 21009 raft_consensus.cc:348] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.078173 21009 raft_consensus.cc:374] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.078259 21009 raft_consensus.cc:729] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.079638 20997 tablet_bootstrap.cc:492] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:56.079715 20995 tablet_bootstrap.cc:492] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:56.079736 20997 ts_tablet_manager.cc:1367] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240501 14:09:56.079792 20995 ts_tablet_manager.cc:1367] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:56.079921 21007 ts_tablet_manager.cc:1398] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:09:56.080173 20995 raft_consensus.cc:348] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.080252 20995 raft_consensus.cc:374] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.080273 20995 raft_consensus.cc:729] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.080240 20997 raft_consensus.cc:348] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.080319 20997 raft_consensus.cc:374] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.080338 20997 raft_consensus.cc:729] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.080367 20995 consensus_queue.cc:260] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.080437 20997 consensus_queue.cc:260] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.080664 20995 ts_tablet_manager.cc:1398] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.080713 20997 ts_tablet_manager.cc:1398] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.080832 20997 tablet_bootstrap.cc:492] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:56.079345 21009 consensus_queue.cc:260] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.081117 21009 ts_tablet_manager.cc:1398] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:56.081281 21017 tablet_bootstrap.cc:492] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:56.081602 20997 tablet_bootstrap.cc:654] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.082079 21017 tablet_bootstrap.cc:654] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.087128 21017 tablet_bootstrap.cc:492] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:56.087222 21017 ts_tablet_manager.cc:1367] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:56.087633 21017 raft_consensus.cc:348] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.087800 21017 raft_consensus.cc:374] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.087888 21017 raft_consensus.cc:729] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.088191 21017 consensus_queue.cc:260] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.088558 21017 ts_tablet_manager.cc:1398] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.089080 20997 tablet_bootstrap.cc:492] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:56.089157 20997 ts_tablet_manager.cc:1367] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:09:56.089522 20997 raft_consensus.cc:348] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.089645 20997 raft_consensus.cc:374] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.090008 21008 tablet_bootstrap.cc:492] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:56.090114 21008 ts_tablet_manager.cc:1367] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20240501 14:09:56.091845 21008 raft_consensus.cc:348] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.091960 21008 raft_consensus.cc:374] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.091996 21008 raft_consensus.cc:729] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.092149 21008 consensus_queue.cc:260] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.092443 21008 ts_tablet_manager.cc:1398] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.092621 20997 raft_consensus.cc:729] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.092799 20997 consensus_queue.cc:260] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.093122 20997 ts_tablet_manager.cc:1398] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:56.101647 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:56.105276 12584 raft_consensus.cc:1261] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.105273 15221 raft_consensus.cc:1261] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:56.105695 20777 consensus_queue.cc:1035] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.105890 21003 consensus_queue.cc:1035] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.108616 20890 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:56.108899 20890 tablet.cc:1620] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:56.110066 12443 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]). Expected version 1 got 0
I20240501 14:09:56.110463 12440 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]). Expected version 1 got 0
I20240501 14:09:56.110898 12438 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]). Expected version 1 got 0
I20240501 14:09:56.111946 21023 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:56.112417 21023 tablet.cc:1620] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:56.113813 12443 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]). Expected version 1 got 0
I20240501 14:09:56.114024 21025 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:56.114300 21025 tablet.cc:1620] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:56.117609 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9] alter complete (version 1)
I20240501 14:09:56.117789 12440 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9]). Expected version 1 got 0
I20240501 14:09:56.146497 21016 raft_consensus.cc:482] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.146631 21016 raft_consensus.cc:504] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.146998 21016 leader_election.cc:288] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:56.147382 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b32701a9b1847c6b81a603332f1e1c0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:56.147521 12584 raft_consensus.cc:2454] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:56.147496 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b32701a9b1847c6b81a603332f1e1c0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:56.147632 12717 raft_consensus.cc:2454] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:56.147861 15288 leader_election.cc:302] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:56.148106 21015 raft_consensus.cc:2781] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.148170 21015 raft_consensus.cc:482] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.148217 21015 raft_consensus.cc:3037] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.149417 21015 raft_consensus.cc:504] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:09:56.149760 21015 leader_election.cc:288] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:56.150185 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b32701a9b1847c6b81a603332f1e1c0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:56.150331 12717 raft_consensus.cc:3037] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.150358 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b32701a9b1847c6b81a603332f1e1c0" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:56.150451 12584 raft_consensus.cc:3037] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.151356 12717 raft_consensus.cc:2454] T 0b32701a9b1847c6b81a603332f1e1c0 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:56.151535 12584 raft_consensus.cc:2454] T 0b32701a9b1847c6b81a603332f1e1c0 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:56.151690 15286 leader_election.cc:302] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:56.152297 21015 raft_consensus.cc:2781] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.152375 21015 raft_consensus.cc:686] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:56.152504 21015 consensus_queue.cc:237] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
W20240501 14:09:56.153431 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:56.154889 20152 raft_consensus.cc:482] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.154985 20152 raft_consensus.cc:504] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.155143 12438 catalog_manager.cc:5484] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:56.155390 20152 leader_election.cc:288] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:56.155989 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8493cd6b3416ab87340a662b4126d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:56.156024 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8493cd6b3416ab87340a662b4126d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:56.156139 14958 raft_consensus.cc:2454] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:56.156139 12848 raft_consensus.cc:2454] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:56.156636 15155 leader_election.cc:302] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:56.157332 20152 raft_consensus.cc:2781] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.157399 20152 raft_consensus.cc:482] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.157438 20152 raft_consensus.cc:3037] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.158344 20152 raft_consensus.cc:504] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.158670 20152 leader_election.cc:288] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:56.159068 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8493cd6b3416ab87340a662b4126d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:56.159184 12848 raft_consensus.cc:3037] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.159919 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8493cd6b3416ab87340a662b4126d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:56.160072 14958 raft_consensus.cc:3037] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.161008 14958 raft_consensus.cc:2454] T 19c8493cd6b3416ab87340a662b4126d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:56.161998 15155 leader_election.cc:302] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:56.162158 20152 raft_consensus.cc:2781] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.162249 20152 raft_consensus.cc:686] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:56.162374 20152 consensus_queue.cc:237] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.162878 12848 raft_consensus.cc:2454] T 19c8493cd6b3416ab87340a662b4126d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:56.164714 12438 catalog_manager.cc:5484] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:56.195855 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:56.221341 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:56.227340 20890 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:56.227708 20890 tablet.cc:1620] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:56.227878 21025 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:56.228176 21025 tablet.cc:1620] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:56.229609 21023 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:56.229887 21023 tablet.cc:1620] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:09:56.235350 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=00ce3c06156f4dd8a94939764e4bb4a9] alter complete (version 2)
W20240501 14:09:56.307130 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:56.323518 20858 raft_consensus.cc:482] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.323632 20858 raft_consensus.cc:504] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.323961 20858 leader_election.cc:288] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:56.324265 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab177cf70c34fc5b6a931b175a14e73" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:56.324409 14957 raft_consensus.cc:2454] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:56.324388 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab177cf70c34fc5b6a931b175a14e73" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:09:56.324509 15220 raft_consensus.cc:2454] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:56.324676 12914 leader_election.cc:302] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:56.324893 20858 raft_consensus.cc:2781] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.324955 20858 raft_consensus.cc:482] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.324983 20858 raft_consensus.cc:3037] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.325827 20858 raft_consensus.cc:504] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.326143 20858 leader_election.cc:288] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:09:56.326459 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab177cf70c34fc5b6a931b175a14e73" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:09:56.326591 15220 raft_consensus.cc:3037] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.326570 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab177cf70c34fc5b6a931b175a14e73" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:56.326676 14957 raft_consensus.cc:3037] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.327626 14957 raft_consensus.cc:2454] T aab177cf70c34fc5b6a931b175a14e73 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:56.327626 15220 raft_consensus.cc:2454] T aab177cf70c34fc5b6a931b175a14e73 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:56.327905 12914 leader_election.cc:302] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:56.328171 20798 raft_consensus.cc:2781] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.328321 20798 raft_consensus.cc:686] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:56.328460 20798 consensus_queue.cc:237] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:56.330682 12438 catalog_manager.cc:5484] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:56.340448 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:56.345587 20890 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.346483 21023 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.347925 21025 tablet.cc:1722] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:56.353257 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=00ce3c06156f4dd8a94939764e4bb4a9] alter complete (version 3)
W20240501 14:09:56.373710 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:56.454572 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:56.454722 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:56.456918 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 4fc1b298254c4a54868e28913abfeb85
I20240501 14:09:56.457484 15202 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc1b298254c4a54868e28913abfeb85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:56.457497 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc1b298254c4a54868e28913abfeb85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:09:56.457845 21053 tablet_replica.cc:330] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:09:56.457852 21052 tablet_replica.cc:330] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:56.457909 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc1b298254c4a54868e28913abfeb85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:56 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:56.458006 21052 raft_consensus.cc:2229] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.458034 21053 raft_consensus.cc:2229] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:56.458212 21052 raft_consensus.cc:2258] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.458262 21054 tablet_replica.cc:330] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:56.458269 21053 raft_consensus.cc:2258] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.458654 21054 raft_consensus.cc:2229] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:56.458885 21053 ts_tablet_manager.cc:1875] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.458923 21052 ts_tablet_manager.cc:1875] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.458922 21054 raft_consensus.cc:2258] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:56.459472 21054 ts_tablet_manager.cc:1875] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:56.460848 21053 ts_tablet_manager.cc:1888] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.460934 21053 log.cc:1196] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/4fc1b298254c4a54868e28913abfeb85
I20240501 14:09:56.461257 21053 ts_tablet_manager.cc:1909] T 4fc1b298254c4a54868e28913abfeb85 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:09:56.461376 21052 ts_tablet_manager.cc:1888] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.461477 21052 log.cc:1196] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/4fc1b298254c4a54868e28913abfeb85
I20240501 14:09:56.461819 21052 ts_tablet_manager.cc:1909] T 4fc1b298254c4a54868e28913abfeb85 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:56.461822 21054 ts_tablet_manager.cc:1888] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:56.461884 21054 log.cc:1196] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/4fc1b298254c4a54868e28913abfeb85
I20240501 14:09:56.462013 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table_new_name [id=00ce3c06156f4dd8a94939764e4bb4a9]) successfully deleted
I20240501 14:09:56.462155 21054 ts_tablet_manager.cc:1909] T 4fc1b298254c4a54868e28913abfeb85 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:56.462814 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table_new_name [id=00ce3c06156f4dd8a94939764e4bb4a9]) successfully deleted
I20240501 14:09:56.462853 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 4fc1b298254c4a54868e28913abfeb85 (table rebalancer_dynamic_table_new_name [id=00ce3c06156f4dd8a94939764e4bb4a9]) successfully deleted
W20240501 14:09:56.465627 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:56.556681 20152 consensus_queue.cc:1035] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.559487 20152 consensus_queue.cc:1035] T 19c8493cd6b3416ab87340a662b4126d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.560253 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:56.566488 12827 tablet_service.cc:1467] Processing CreateTablet for tablet ddc54278698248febd2301e61f8e9ec7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]), partition=
I20240501 14:09:56.566704 12959 tablet_service.cc:1467] Processing CreateTablet for tablet ddc54278698248febd2301e61f8e9ec7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]), partition=
I20240501 14:09:56.566814 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc54278698248febd2301e61f8e9ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.566987 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc54278698248febd2301e61f8e9ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.568634 21007 tablet_bootstrap.cc:492] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:56.569090 12696 tablet_service.cc:1467] Processing CreateTablet for tablet ddc54278698248febd2301e61f8e9ec7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]), partition=
I20240501 14:09:56.569329 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc54278698248febd2301e61f8e9ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.569588 21007 tablet_bootstrap.cc:654] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.570210 21009 tablet_bootstrap.cc:492] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:56.571156 21009 tablet_bootstrap.cc:654] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.571600 21008 tablet_bootstrap.cc:492] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:56.572481 21008 tablet_bootstrap.cc:654] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.572808 21007 tablet_bootstrap.cc:492] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:56.572872 21007 ts_tablet_manager.cc:1367] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:56.573344 21007 raft_consensus.cc:348] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.573439 21007 raft_consensus.cc:374] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.573472 21007 raft_consensus.cc:729] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.573581 21008 tablet_bootstrap.cc:492] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:56.573580 21009 tablet_bootstrap.cc:492] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:56.573606 21007 consensus_queue.cc:260] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.573657 21009 ts_tablet_manager.cc:1367] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:56.573674 21008 ts_tablet_manager.cc:1367] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:09:56.573892 21007 ts_tablet_manager.cc:1398] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.574065 21009 raft_consensus.cc:348] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.574066 21008 raft_consensus.cc:348] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.574187 21008 raft_consensus.cc:374] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.574187 21009 raft_consensus.cc:374] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.574227 21009 raft_consensus.cc:729] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.574227 21008 raft_consensus.cc:729] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.574330 21009 consensus_queue.cc:260] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.574347 21008 consensus_queue.cc:260] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.574586 21008 ts_tablet_manager.cc:1398] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:56.574597 21009 ts_tablet_manager.cc:1398] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.591156 21015 consensus_queue.cc:1035] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.594758 21016 consensus_queue.cc:1035] T 0b32701a9b1847c6b81a603332f1e1c0 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.731350 21039 consensus_queue.cc:1035] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:56.735855 21040 consensus_queue.cc:1035] T aab177cf70c34fc5b6a931b175a14e73 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:56.762346 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:56.873143 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:56.881018 15332 tablet_service.cc:1467] Processing CreateTablet for tablet 4193e34dc3a6456eaa7fd3c1d14a4844 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.881341 15332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4193e34dc3a6456eaa7fd3c1d14a4844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.882071 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 4193e34dc3a6456eaa7fd3c1d14a4844 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.882376 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4193e34dc3a6456eaa7fd3c1d14a4844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.883661 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffc53a0d3e141d8a88e890ac12fe098 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.883877 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffc53a0d3e141d8a88e890ac12fe098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.882093 14936 tablet_service.cc:1467] Processing CreateTablet for tablet a07b1c7673684f528b1152d23c5e0821 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.884752 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07b1c7673684f528b1152d23c5e0821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.885102 21090 tablet_bootstrap.cc:492] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:56.885941 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 4193e34dc3a6456eaa7fd3c1d14a4844 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.886189 21089 tablet_bootstrap.cc:492] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:56.886242 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4193e34dc3a6456eaa7fd3c1d14a4844. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.886585 12959 tablet_service.cc:1467] Processing CreateTablet for tablet a07b1c7673684f528b1152d23c5e0821 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.886869 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07b1c7673684f528b1152d23c5e0821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.887305 21089 tablet_bootstrap.cc:654] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.889010 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffc53a0d3e141d8a88e890ac12fe098 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.889106 21009 tablet_bootstrap.cc:492] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:56.889304 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffc53a0d3e141d8a88e890ac12fe098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.889330 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 6ffc53a0d3e141d8a88e890ac12fe098 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.889644 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ffc53a0d3e141d8a88e890ac12fe098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.890118 21009 tablet_bootstrap.cc:654] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.891170 21090 tablet_bootstrap.cc:654] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.891831 21008 tablet_bootstrap.cc:492] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:56.892014 12566 tablet_service.cc:1467] Processing CreateTablet for tablet a07b1c7673684f528b1152d23c5e0821 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=596f71d9c46d46888236d37156ca1b83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:56.892143 21094 tablet_bootstrap.cc:492] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:56.892279 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a07b1c7673684f528b1152d23c5e0821. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:56.893002 21008 tablet_bootstrap.cc:654] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.893101 21009 tablet_bootstrap.cc:492] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:56.893194 21009 ts_tablet_manager.cc:1367] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:56.893682 21009 raft_consensus.cc:348] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:56.893811 21009 raft_consensus.cc:374] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.893851 21009 raft_consensus.cc:729] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.893996 21009 consensus_queue.cc:260] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:56.894297 21009 ts_tablet_manager.cc:1398] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.895726 21008 tablet_bootstrap.cc:492] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:56.895812 21008 ts_tablet_manager.cc:1367] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:56.896312 21008 raft_consensus.cc:348] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.896426 21008 raft_consensus.cc:374] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.896458 21008 raft_consensus.cc:729] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.896560 21008 consensus_queue.cc:260] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.896826 21008 ts_tablet_manager.cc:1398] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:56.897028 21094 tablet_bootstrap.cc:654] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.897223 21092 tablet_bootstrap.cc:492] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:56.897543 21089 tablet_bootstrap.cc:492] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:56.897682 21089 ts_tablet_manager.cc:1367] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:56.898125 21089 raft_consensus.cc:348] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:56.898285 21089 raft_consensus.cc:374] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.898298 21092 tablet_bootstrap.cc:654] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.898381 21089 raft_consensus.cc:729] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.898564 21089 consensus_queue.cc:260] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:56.899027 21089 ts_tablet_manager.cc:1398] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.899196 21090 tablet_bootstrap.cc:492] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:56.899295 21090 ts_tablet_manager.cc:1367] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:09:56.899770 21090 raft_consensus.cc:348] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:56.899873 21090 raft_consensus.cc:374] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.899924 21090 raft_consensus.cc:729] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.900061 21090 consensus_queue.cc:260] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:56.900336 21089 tablet_bootstrap.cc:492] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:56.900341 21090 ts_tablet_manager.cc:1398] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.900517 21090 tablet_bootstrap.cc:492] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:56.901293 21089 tablet_bootstrap.cc:654] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.901408 21090 tablet_bootstrap.cc:654] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.903730 21094 tablet_bootstrap.cc:492] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:56.903843 21090 tablet_bootstrap.cc:492] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:56.903884 21094 ts_tablet_manager.cc:1367] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:09:56.903970 21090 ts_tablet_manager.cc:1367] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:56.904395 21094 raft_consensus.cc:348] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.904568 21094 raft_consensus.cc:374] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.904523 21090 raft_consensus.cc:348] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:56.904618 21090 raft_consensus.cc:374] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.904660 21094 raft_consensus.cc:729] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.904706 21090 raft_consensus.cc:729] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.904857 21094 consensus_queue.cc:260] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.904929 21090 consensus_queue.cc:260] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:56.905236 21094 ts_tablet_manager.cc:1398] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.905324 21090 ts_tablet_manager.cc:1398] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.906536 21089 tablet_bootstrap.cc:492] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:56.906955 21089 ts_tablet_manager.cc:1367] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:56.907442 21089 raft_consensus.cc:348] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.907609 21089 raft_consensus.cc:374] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.907680 21089 raft_consensus.cc:729] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.907845 21089 consensus_queue.cc:260] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.908355 21089 ts_tablet_manager.cc:1398] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.910193 20361 raft_consensus.cc:482] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:56.910290 20361 raft_consensus.cc:504] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.910706 20361 leader_election.cc:288] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:56.912233 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc54278698248febd2301e61f8e9ec7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:56.912366 12978 raft_consensus.cc:2454] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:56.912811 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc54278698248febd2301e61f8e9ec7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:56.912945 12716 raft_consensus.cc:2454] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:56.913197 12782 leader_election.cc:302] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:56.913913 20961 raft_consensus.cc:2781] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:56.913977 20961 raft_consensus.cc:482] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:56.914017 20961 raft_consensus.cc:3037] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.915009 20961 raft_consensus.cc:504] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.915325 20961 leader_election.cc:288] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:56.915779 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc54278698248febd2301e61f8e9ec7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:56.915906 12716 raft_consensus.cc:3037] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.915877 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc54278698248febd2301e61f8e9ec7" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:56.915992 12978 raft_consensus.cc:3037] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:56.916841 12978 raft_consensus.cc:2454] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:56.916849 12716 raft_consensus.cc:2454] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:56.917106 12784 leader_election.cc:302] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:56.917385 20361 raft_consensus.cc:2781] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:56.917541 20361 raft_consensus.cc:686] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:56.917695 20361 consensus_queue.cc:237] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:56.920253 12438 catalog_manager.cc:5484] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:56.923231 21094 tablet_bootstrap.cc:492] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:56.924228 21094 tablet_bootstrap.cc:654] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:56.925460 21094 tablet_bootstrap.cc:492] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:56.925572 21094 ts_tablet_manager.cc:1367] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:56.925776 21092 tablet_bootstrap.cc:492] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:56.925873 21092 ts_tablet_manager.cc:1367] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240501 14:09:56.926057 21094 raft_consensus.cc:348] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:56.926174 21094 raft_consensus.cc:374] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.926223 21094 raft_consensus.cc:729] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.926357 21094 consensus_queue.cc:260] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:56.926376 21092 raft_consensus.cc:348] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:56.926470 21092 raft_consensus.cc:374] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:56.926510 21092 raft_consensus.cc:729] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:56.926635 21092 consensus_queue.cc:260] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:56.926687 21094 ts_tablet_manager.cc:1398] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:56.926919 21092 ts_tablet_manager.cc:1398] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:56.968463 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.015614 21104 raft_consensus.cc:482] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.015743 21104 raft_consensus.cc:504] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.016211 21104 leader_election.cc:288] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:57.016559 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffc53a0d3e141d8a88e890ac12fe098" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:57.016721 12584 raft_consensus.cc:2454] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:57.016732 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffc53a0d3e141d8a88e890ac12fe098" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:57.016889 12716 raft_consensus.cc:2454] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:09:57.016996 15288 leader_election.cc:302] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:57.017165 21104 raft_consensus.cc:2781] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.017210 21104 raft_consensus.cc:482] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.017254 21104 raft_consensus.cc:3037] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.018301 21104 raft_consensus.cc:504] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.018853 21104 leader_election.cc:288] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
W20240501 14:09:57.019191 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:57.019340 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffc53a0d3e141d8a88e890ac12fe098" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:57.019467 12716 raft_consensus.cc:3037] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.020196 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ffc53a0d3e141d8a88e890ac12fe098" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:57.020285 12584 raft_consensus.cc:3037] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.020668 12716 raft_consensus.cc:2454] T 6ffc53a0d3e141d8a88e890ac12fe098 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:57.021147 12584 raft_consensus.cc:2454] T 6ffc53a0d3e141d8a88e890ac12fe098 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:09:57.021531 15288 leader_election.cc:302] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:57.021695 21016 raft_consensus.cc:2781] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.021773 21016 raft_consensus.cc:686] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:09:57.021906 21016 consensus_queue.cc:237] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.024758 12438 catalog_manager.cc:5484] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:57.033828 20152 raft_consensus.cc:482] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.033967 20152 raft_consensus.cc:504] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.034377 20152 leader_election.cc:288] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:57.034873 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4193e34dc3a6456eaa7fd3c1d14a4844" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:57.034930 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4193e34dc3a6456eaa7fd3c1d14a4844" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:57.035027 14958 raft_consensus.cc:2454] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:57.035089 15351 raft_consensus.cc:2454] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:57.035372 15155 leader_election.cc:302] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:57.036165 20152 raft_consensus.cc:2781] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.036226 20152 raft_consensus.cc:482] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.036265 20152 raft_consensus.cc:3037] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:57.036633 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.037446 20152 raft_consensus.cc:504] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.037829 20152 leader_election.cc:288] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:57.038416 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4193e34dc3a6456eaa7fd3c1d14a4844" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:57.038565 14958 raft_consensus.cc:3037] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.039639 14958 raft_consensus.cc:2454] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:57.040103 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4193e34dc3a6456eaa7fd3c1d14a4844" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:57.040238 15351 raft_consensus.cc:3037] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.041193 15351 raft_consensus.cc:2454] T 4193e34dc3a6456eaa7fd3c1d14a4844 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:57.041895 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:57.043176 15155 leader_election.cc:302] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:57.043895 21056 raft_consensus.cc:2781] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.043987 21056 raft_consensus.cc:686] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:57.044126 21056 consensus_queue.cc:237] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.046504 12716 raft_consensus.cc:1261] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.047061 20961 consensus_queue.cc:1035] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.048134 12978 raft_consensus.cc:1261] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.048566 20961 consensus_queue.cc:1035] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.049481 12438 catalog_manager.cc:5484] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:57.050769 12443 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]). Expected version 1 got 0
I20240501 14:09:57.052053 12443 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]). Expected version 1 got 0
W20240501 14:09:57.053890 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.056700 21119 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.057157 21119 tablet.cc:1620] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:57.057842 12438 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]). Expected version 1 got 0
I20240501 14:09:57.058885 21120 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.059652 12438 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31]). Expected version 1 got 0
I20240501 14:09:57.060994 21120 tablet.cc:1620] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:57.063735 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31] alter complete (version 1)
W20240501 14:09:57.069248 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:09:57.068997 21121 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.069541 21121 tablet.cc:1620] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
W20240501 14:09:57.084281 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.166289 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:57.172052 21119 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:57.172410 21119 tablet.cc:1620] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:57.172633 21120 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:57.172920 21120 tablet.cc:1620] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:57.173075 21121 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:57.173436 21121 tablet.cc:1620] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:57.178464 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=cefe6da70d9647be8570be1f62af8d31] alter complete (version 2)
W20240501 14:09:57.194886 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:57.225369 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.234287 21122 raft_consensus.cc:482] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.234616 21122 raft_consensus.cc:504] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:57.235190 21122 leader_election.cc:288] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:57.235601 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07b1c7673684f528b1152d23c5e0821" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:57.235661 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07b1c7673684f528b1152d23c5e0821" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:57.235767 12978 raft_consensus.cc:2454] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:57.235796 14958 raft_consensus.cc:2454] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:57.236135 12522 leader_election.cc:302] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:57.236480 21122 raft_consensus.cc:2781] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.236559 21122 raft_consensus.cc:482] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.236595 21122 raft_consensus.cc:3037] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.237790 21122 raft_consensus.cc:504] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:57.238307 21122 leader_election.cc:288] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:57.238831 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07b1c7673684f528b1152d23c5e0821" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:57.238960 14958 raft_consensus.cc:3037] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.238773 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a07b1c7673684f528b1152d23c5e0821" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:57.239207 12978 raft_consensus.cc:3037] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.240173 14958 raft_consensus.cc:2454] T a07b1c7673684f528b1152d23c5e0821 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:57.240437 12978 raft_consensus.cc:2454] T a07b1c7673684f528b1152d23c5e0821 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:57.240583 12521 leader_election.cc:302] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:57.241472 21122 raft_consensus.cc:2781] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.241608 21122 raft_consensus.cc:686] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:57.241776 21122 consensus_queue.cc:237] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:09:57.245478 12443 catalog_manager.cc:5484] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:57.261343 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:57.267952 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.283432 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:57.288889 21119 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:57.289296 21120 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:57.290833 21121 tablet.cc:1722] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:57.295619 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=cefe6da70d9647be8570be1f62af8d31] alter complete (version 3)
W20240501 14:09:57.302175 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:09:57.340440 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:57.341045 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:57.366658 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:57.370622 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.401244 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:57.401402 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:57.404027 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet ddc54278698248febd2301e61f8e9ec7
I20240501 14:09:57.404898 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc54278698248febd2301e61f8e9ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:57 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:57.405330 21134 tablet_replica.cc:330] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:57.405568 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc54278698248febd2301e61f8e9ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:57 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:57.405702 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc54278698248febd2301e61f8e9ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:57 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:57.405869 21135 tablet_replica.cc:330] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:57.406004 21135 raft_consensus.cc:2229] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:57.406212 21135 raft_consensus.cc:2258] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:57.406795 21135 ts_tablet_manager.cc:1875] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:57.406854 21134 raft_consensus.cc:2229] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:57.407028 21134 raft_consensus.cc:2258] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:57.407722 21134 ts_tablet_manager.cc:1875] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:57.409150 21135 ts_tablet_manager.cc:1888] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:57.409221 21135 log.cc:1196] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/ddc54278698248febd2301e61f8e9ec7
I20240501 14:09:57.409548 21135 ts_tablet_manager.cc:1909] T ddc54278698248febd2301e61f8e9ec7 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:57.410214 21134 ts_tablet_manager.cc:1888] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:57.410295 21134 log.cc:1196] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/ddc54278698248febd2301e61f8e9ec7
I20240501 14:09:57.410411 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table_new_name [id=cefe6da70d9647be8570be1f62af8d31]) successfully deleted
I20240501 14:09:57.410807 21134 ts_tablet_manager.cc:1909] T ddc54278698248febd2301e61f8e9ec7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:57.410923 21136 tablet_replica.cc:330] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:57.411135 21136 raft_consensus.cc:2229] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:57.411288 21136 raft_consensus.cc:2258] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:57.411909 21136 ts_tablet_manager.cc:1875] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:57.412817 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table_new_name [id=cefe6da70d9647be8570be1f62af8d31]) successfully deleted
I20240501 14:09:57.414186 21136 ts_tablet_manager.cc:1888] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:57.414255 21136 log.cc:1196] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/ddc54278698248febd2301e61f8e9ec7
I20240501 14:09:57.414548 21136 ts_tablet_manager.cc:1909] T ddc54278698248febd2301e61f8e9ec7 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:57.415335 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet ddc54278698248febd2301e61f8e9ec7 (table rebalancer_dynamic_table_new_name [id=cefe6da70d9647be8570be1f62af8d31]) successfully deleted
I20240501 14:09:57.421953 21015 consensus_queue.cc:1035] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:57.422667 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:57.425900 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.428555 21104 consensus_queue.cc:1035] T 6ffc53a0d3e141d8a88e890ac12fe098 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:57.487859 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:09:57.502092 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.505906 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:57.511932 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 3f21fba71e41409ba05f067db7ada9a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]), partition=
I20240501 14:09:57.511932 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 3f21fba71e41409ba05f067db7ada9a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]), partition=
I20240501 14:09:57.512215 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f21fba71e41409ba05f067db7ada9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.512215 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f21fba71e41409ba05f067db7ada9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.514494 21152 tablet_bootstrap.cc:492] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:57.514887 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 3f21fba71e41409ba05f067db7ada9a5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]), partition=
I20240501 14:09:57.515139 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f21fba71e41409ba05f067db7ada9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.515655 21152 tablet_bootstrap.cc:654] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
W20240501 14:09:57.516338 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.516784 21151 tablet_bootstrap.cc:492] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:57.517930 21151 tablet_bootstrap.cc:654] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.518386 21152 tablet_bootstrap.cc:492] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:57.518550 21152 ts_tablet_manager.cc:1367] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:57.519152 21152 raft_consensus.cc:348] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.519304 21152 raft_consensus.cc:374] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.519367 21152 raft_consensus.cc:729] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.519481 21154 tablet_bootstrap.cc:492] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:57.519580 21152 consensus_queue.cc:260] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.519932 21152 ts_tablet_manager.cc:1398] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.520968 21154 tablet_bootstrap.cc:654] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.522118 21154 tablet_bootstrap.cc:492] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:57.522238 21154 ts_tablet_manager.cc:1367] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:57.522265 21151 tablet_bootstrap.cc:492] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:57.522343 21151 ts_tablet_manager.cc:1367] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:57.522739 21154 raft_consensus.cc:348] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.522887 21154 raft_consensus.cc:374] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.522835 21151 raft_consensus.cc:348] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.522919 21154 raft_consensus.cc:729] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.522935 21151 raft_consensus.cc:374] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.522971 21151 raft_consensus.cc:729] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.523108 21154 consensus_queue.cc:260] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.523118 21151 consensus_queue.cc:260] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.523412 21151 ts_tablet_manager.cc:1398] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.524204 21154 ts_tablet_manager.cc:1398] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:57.593330 20152 consensus_queue.cc:1035] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.597012 21056 consensus_queue.cc:1035] T 4193e34dc3a6456eaa7fd3c1d14a4844 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:57.619280 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:09:57.619988 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:57.685154 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:57.686568 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:09:57.696507 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.747476 21038 consensus_queue.cc:1035] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.751474 21174 consensus_queue.cc:1035] T a07b1c7673684f528b1152d23c5e0821 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.763196 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:57.769447 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 5234773677904755ae9175c818651541 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.769575 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 5234773677904755ae9175c818651541 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.769812 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5234773677904755ae9175c818651541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.769810 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5234773677904755ae9175c818651541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.770551 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 24297372f37146f0b5bb5d9a2d633f37 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.770663 12828 tablet_service.cc:1467] Processing CreateTablet for tablet d3c95fa8e068426fab380bbd3b9ff19a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.770855 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24297372f37146f0b5bb5d9a2d633f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.770936 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c95fa8e068426fab380bbd3b9ff19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.771139 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 24297372f37146f0b5bb5d9a2d633f37 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.771390 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24297372f37146f0b5bb5d9a2d633f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.772868 21151 tablet_bootstrap.cc:492] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:57.773492 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 5234773677904755ae9175c818651541 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.773708 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5234773677904755ae9175c818651541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.773967 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 24297372f37146f0b5bb5d9a2d633f37 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.774042 21151 tablet_bootstrap.cc:654] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.774230 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24297372f37146f0b5bb5d9a2d633f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.774466 21176 tablet_bootstrap.cc:492] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:57.775106 15331 tablet_service.cc:1467] Processing CreateTablet for tablet d3c95fa8e068426fab380bbd3b9ff19a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.775347 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c95fa8e068426fab380bbd3b9ff19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.775971 15201 tablet_service.cc:1467] Processing CreateTablet for tablet d3c95fa8e068426fab380bbd3b9ff19a (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=8577c7c9cb1f4f0894196d8a1e7d95ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:57.776165 21176 tablet_bootstrap.cc:654] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.776212 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c95fa8e068426fab380bbd3b9ff19a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:57.777154 21179 tablet_bootstrap.cc:492] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:57.777899 21151 tablet_bootstrap.cc:492] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:57.777978 21151 ts_tablet_manager.cc:1367] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:57.778194 21179 tablet_bootstrap.cc:654] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.778414 21151 raft_consensus.cc:348] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.778532 21151 raft_consensus.cc:374] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.778566 21151 raft_consensus.cc:729] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.778751 21151 consensus_queue.cc:260] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.779028 21151 ts_tablet_manager.cc:1398] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.779198 21151 tablet_bootstrap.cc:492] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:57.780027 21151 tablet_bootstrap.cc:654] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.782011 21176 tablet_bootstrap.cc:492] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:57.782091 21176 ts_tablet_manager.cc:1367] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:57.782502 21176 raft_consensus.cc:348] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.782721 21176 raft_consensus.cc:374] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.782800 21176 raft_consensus.cc:729] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.782936 21176 consensus_queue.cc:260] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.783052 21151 tablet_bootstrap.cc:492] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:57.783210 21176 ts_tablet_manager.cc:1398] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:57.783353 21176 tablet_bootstrap.cc:492] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:57.784196 21176 tablet_bootstrap.cc:654] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.784904 21151 ts_tablet_manager.cc:1367] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:57.785378 21151 raft_consensus.cc:348] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.785473 21151 raft_consensus.cc:374] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.785509 21151 raft_consensus.cc:729] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.785640 21151 consensus_queue.cc:260] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.785935 21151 ts_tablet_manager.cc:1398] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.787717 21151 tablet_bootstrap.cc:492] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:57.788408 21176 tablet_bootstrap.cc:492] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:57.788503 21176 ts_tablet_manager.cc:1367] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 14:09:57.789038 21176 raft_consensus.cc:348] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.789182 21176 raft_consensus.cc:374] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.789239 21176 raft_consensus.cc:729] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.789371 21176 consensus_queue.cc:260] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.789767 21176 ts_tablet_manager.cc:1398] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:57.790343 21151 tablet_bootstrap.cc:654] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.791270 21183 tablet_bootstrap.cc:492] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:57.792515 21151 tablet_bootstrap.cc:492] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:57.792601 21151 ts_tablet_manager.cc:1367] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:57.793114 21177 tablet_bootstrap.cc:492] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:57.793330 21151 raft_consensus.cc:348] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.793435 21151 raft_consensus.cc:374] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.793466 21151 raft_consensus.cc:729] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.793641 21151 consensus_queue.cc:260] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.794025 21151 ts_tablet_manager.cc:1398] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.794584 21183 tablet_bootstrap.cc:654] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.795874 21177 tablet_bootstrap.cc:654] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.797610 21183 tablet_bootstrap.cc:492] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:57.797844 21183 ts_tablet_manager.cc:1367] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:57.798285 21183 raft_consensus.cc:348] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.798393 21183 raft_consensus.cc:374] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.798425 21183 raft_consensus.cc:729] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.798599 21183 consensus_queue.cc:260] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.799108 21183 ts_tablet_manager.cc:1398] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.799925 21179 tablet_bootstrap.cc:492] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:57.800009 21179 ts_tablet_manager.cc:1367] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:09:57.800062 21177 tablet_bootstrap.cc:492] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:57.800146 21177 ts_tablet_manager.cc:1367] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:57.800453 21179 raft_consensus.cc:348] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.800572 21179 raft_consensus.cc:374] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.800608 21179 raft_consensus.cc:729] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.800572 21177 raft_consensus.cc:348] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.800716 21179 consensus_queue.cc:260] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.800760 21177 raft_consensus.cc:374] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.800984 21177 raft_consensus.cc:729] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.801010 21179 ts_tablet_manager.cc:1398] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:57.804785 20961 raft_consensus.cc:482] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.804872 20961 raft_consensus.cc:504] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.805346 20961 leader_election.cc:288] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:57.805986 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5234773677904755ae9175c818651541" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:57.806097 12585 raft_consensus.cc:2454] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:57.806360 12784 leader_election.cc:302] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:09:57.806533 20961 raft_consensus.cc:2781] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.806571 20961 raft_consensus.cc:482] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.806519 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5234773677904755ae9175c818651541" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:57.806602 20961 raft_consensus.cc:3037] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.806648 15352 raft_consensus.cc:2454] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:57.807626 20961 raft_consensus.cc:504] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.807966 20961 leader_election.cc:288] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:57.808245 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5234773677904755ae9175c818651541" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:57.808362 12585 raft_consensus.cc:3037] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.808326 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5234773677904755ae9175c818651541" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:57.808429 15352 raft_consensus.cc:3037] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.809172 15352 raft_consensus.cc:2454] T 5234773677904755ae9175c818651541 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:57.809243 12585 raft_consensus.cc:2454] T 5234773677904755ae9175c818651541 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:57.809562 12785 leader_election.cc:302] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:57.809912 20961 raft_consensus.cc:2781] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.809995 20961 raft_consensus.cc:686] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:57.810115 20961 consensus_queue.cc:237] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.813038 12443 catalog_manager.cc:5484] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:57.801174 21177 consensus_queue.cc:260] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.816095 21177 ts_tablet_manager.cc:1398] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240501 14:09:57.816275 21177 tablet_bootstrap.cc:492] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:57.816382 20961 raft_consensus.cc:482] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.816470 20961 raft_consensus.cc:504] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.816888 20961 leader_election.cc:288] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:57.817211 21177 tablet_bootstrap.cc:654] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:57.817297 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f21fba71e41409ba05f067db7ada9a5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:57.817428 14958 raft_consensus.cc:2454] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:57.817507 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f21fba71e41409ba05f067db7ada9a5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:57.817636 12978 raft_consensus.cc:2454] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:57.817962 12783 leader_election.cc:302] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:57.818118 20961 raft_consensus.cc:2781] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.818166 20961 raft_consensus.cc:482] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.818210 20961 raft_consensus.cc:3037] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.819700 21177 tablet_bootstrap.cc:492] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:57.819785 21177 ts_tablet_manager.cc:1367] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:57.819973 20961 raft_consensus.cc:504] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.820299 20961 leader_election.cc:288] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:57.820264 21177 raft_consensus.cc:348] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.820360 21177 raft_consensus.cc:374] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:57.820392 21177 raft_consensus.cc:729] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:57.820530 21177 consensus_queue.cc:260] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.820652 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f21fba71e41409ba05f067db7ada9a5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:57.820691 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f21fba71e41409ba05f067db7ada9a5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:57.820761 12978 raft_consensus.cc:3037] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.820770 21177 ts_tablet_manager.cc:1398] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:57.820811 14958 raft_consensus.cc:3037] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.821705 12978 raft_consensus.cc:2454] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:57.821705 14958 raft_consensus.cc:2454] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:57.822002 12783 leader_election.cc:302] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:57.822288 20961 raft_consensus.cc:2781] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.823298 20961 raft_consensus.cc:686] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:57.823439 20961 consensus_queue.cc:237] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:09:57.826006 12439 catalog_manager.cc:5484] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:57.829874 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:57.857761 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.916776 21000 raft_consensus.cc:482] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.916922 21000 raft_consensus.cc:504] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.917311 21000 leader_election.cc:288] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:57.917585 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c95fa8e068426fab380bbd3b9ff19a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:57.917750 12849 raft_consensus.cc:2454] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:57.917711 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c95fa8e068426fab380bbd3b9ff19a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:57.917883 15352 raft_consensus.cc:2454] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:57.918216 15157 leader_election.cc:302] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:57.918383 21000 raft_consensus.cc:2781] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.918444 21000 raft_consensus.cc:482] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.918493 21000 raft_consensus.cc:3037] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.919512 21000 raft_consensus.cc:504] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.919857 21000 leader_election.cc:288] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:57.920140 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c95fa8e068426fab380bbd3b9ff19a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:57.920233 12849 raft_consensus.cc:3037] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.921264 12849 raft_consensus.cc:2454] T d3c95fa8e068426fab380bbd3b9ff19a P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:57.921399 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3c95fa8e068426fab380bbd3b9ff19a" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:57.921500 15352 raft_consensus.cc:3037] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.921754 15157 leader_election.cc:302] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:57.922019 21056 raft_consensus.cc:2781] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.922128 21056 raft_consensus.cc:686] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:57.922255 21056 consensus_queue.cc:237] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:57.922462 15352 raft_consensus.cc:2454] T d3c95fa8e068426fab380bbd3b9ff19a P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:57.924674 12439 catalog_manager.cc:5484] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:57.968907 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:57.981665 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:57.984220 21000 raft_consensus.cc:482] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:57.984339 21000 raft_consensus.cc:504] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.984915 21000 leader_election.cc:288] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:57.985404 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24297372f37146f0b5bb5d9a2d633f37" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:57.985404 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24297372f37146f0b5bb5d9a2d633f37" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:57.985538 12717 raft_consensus.cc:2454] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:57.985538 12849 raft_consensus.cc:2454] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:57.986281 14958 raft_consensus.cc:1261] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.986297 12978 raft_consensus.cc:1261] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:57.986636 15157 leader_election.cc:302] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:57.986682 20961 consensus_queue.cc:1035] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.986814 21000 raft_consensus.cc:2781] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:57.986830 20361 consensus_queue.cc:1035] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:57.986873 21000 raft_consensus.cc:482] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:57.986922 21000 raft_consensus.cc:3037] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.987852 21000 raft_consensus.cc:504] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.989190 21205 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.989537 21205 tablet.cc:1620] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:57.990239 21000 leader_election.cc:288] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:57.990391 12439 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]). Expected version 1 got 0
I20240501 14:09:57.990677 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24297372f37146f0b5bb5d9a2d633f37" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:57.990811 12849 raft_consensus.cc:3037] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.991232 12443 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]). Expected version 1 got 0
I20240501 14:09:57.991726 12849 raft_consensus.cc:2454] T 24297372f37146f0b5bb5d9a2d633f37 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:57.992354 15157 leader_election.cc:302] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:09:57.992555 21000 raft_consensus.cc:2781] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:57.992640 21000 raft_consensus.cc:686] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:57.992760 21000 consensus_queue.cc:237] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:57.993870 12716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24297372f37146f0b5bb5d9a2d633f37" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:57.993995 12716 raft_consensus.cc:3037] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:57.995335 12443 catalog_manager.cc:5484] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:57.995484 21208 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:57.995769 21208 tablet.cc:1620] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:57.995790 12438 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]). Expected version 1 got 0
I20240501 14:09:57.996888 12716 raft_consensus.cc:2454] T 24297372f37146f0b5bb5d9a2d633f37 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:58.002528 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3] alter complete (version 1)
I20240501 14:09:58.003038 12438 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3]). Expected version 1 got 0
I20240501 14:09:58.005347 21209 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.005698 21209 tablet.cc:1620] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
W20240501 14:09:58.015425 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:58.107865 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:58.113930 21205 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.114285 21205 tablet.cc:1620] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:58.115049 21208 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.115350 21208 tablet.cc:1620] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:58.117142 21209 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.117424 21209 tablet.cc:1620] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:58.121834 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=ee7fab16e47545ffaba171717c7fbac3] alter complete (version 2)
I20240501 14:09:58.198554 20961 consensus_queue.cc:1035] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.203253 20361 consensus_queue.cc:1035] T 5234773677904755ae9175c818651541 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:58.205571 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:58.224246 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:58.231734 21205 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.232290 21209 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.236759 21208 tablet.cc:1722] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.240083 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=ee7fab16e47545ffaba171717c7fbac3] alter complete (version 3)
I20240501 14:09:58.342650 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:58.342823 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:58.344877 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 3f21fba71e41409ba05f067db7ada9a5
I20240501 14:09:58.345402 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f21fba71e41409ba05f067db7ada9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:58.345515 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f21fba71e41409ba05f067db7ada9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:58.345628 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f21fba71e41409ba05f067db7ada9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:58 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:58.345747 21234 tablet_replica.cc:330] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:58.345867 21234 raft_consensus.cc:2229] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:58.345871 21235 tablet_replica.cc:330] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:58.345969 21236 tablet_replica.cc:330] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:58.346024 21235 raft_consensus.cc:2229] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.346112 21236 raft_consensus.cc:2229] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:58.346122 21234 raft_consensus.cc:2258] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.346163 21235 raft_consensus.cc:2258] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.346437 21236 raft_consensus.cc:2258] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:58.346711 21234 ts_tablet_manager.cc:1875] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.346792 21235 ts_tablet_manager.cc:1875] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.347040 21236 ts_tablet_manager.cc:1875] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:58.348860 21234 ts_tablet_manager.cc:1888] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.348979 21234 log.cc:1196] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f21fba71e41409ba05f067db7ada9a5
I20240501 14:09:58.349222 21235 ts_tablet_manager.cc:1888] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.349253 21236 ts_tablet_manager.cc:1888] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:58.349310 21236 log.cc:1196] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/3f21fba71e41409ba05f067db7ada9a5
I20240501 14:09:58.349310 21235 log.cc:1196] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/3f21fba71e41409ba05f067db7ada9a5
I20240501 14:09:58.349381 21234 ts_tablet_manager.cc:1909] T 3f21fba71e41409ba05f067db7ada9a5 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:58.349607 21235 ts_tablet_manager.cc:1909] T 3f21fba71e41409ba05f067db7ada9a5 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:58.349650 21236 ts_tablet_manager.cc:1909] T 3f21fba71e41409ba05f067db7ada9a5 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:58.350095 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table_new_name [id=ee7fab16e47545ffaba171717c7fbac3]) successfully deleted
I20240501 14:09:58.350289 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table_new_name [id=ee7fab16e47545ffaba171717c7fbac3]) successfully deleted
I20240501 14:09:58.350600 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 3f21fba71e41409ba05f067db7ada9a5 (table rebalancer_dynamic_table_new_name [id=ee7fab16e47545ffaba171717c7fbac3]) successfully deleted
I20240501 14:09:58.350641 21000 consensus_queue.cc:1035] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.354290 21000 consensus_queue.cc:1035] T d3c95fa8e068426fab380bbd3b9ff19a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:58.363904 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:58.415323 21000 consensus_queue.cc:1035] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.421936 21056 consensus_queue.cc:1035] T 24297372f37146f0b5bb5d9a2d633f37 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:58.435639 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:58.446784 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:58.452262 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 0d67e0213a2e406e9e6df4413777e3f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792]), partition=
I20240501 14:09:58.452325 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 0d67e0213a2e406e9e6df4413777e3f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792]), partition=
I20240501 14:09:58.452553 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d67e0213a2e406e9e6df4413777e3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.452592 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d67e0213a2e406e9e6df4413777e3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.452562 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 0d67e0213a2e406e9e6df4413777e3f8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792]), partition=
I20240501 14:09:58.452787 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d67e0213a2e406e9e6df4413777e3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.454486 21243 tablet_bootstrap.cc:492] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:58.454878 21244 tablet_bootstrap.cc:492] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:58.455590 21245 tablet_bootstrap.cc:492] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:58.455830 21243 tablet_bootstrap.cc:654] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.455839 21244 tablet_bootstrap.cc:654] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.456609 21245 tablet_bootstrap.cc:654] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.457013 21243 tablet_bootstrap.cc:492] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:58.457137 21243 ts_tablet_manager.cc:1367] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:58.457818 21244 tablet_bootstrap.cc:492] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:58.457859 21245 tablet_bootstrap.cc:492] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:58.457902 21244 ts_tablet_manager.cc:1367] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:09:58.457947 21245 ts_tablet_manager.cc:1367] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:58.458276 21244 raft_consensus.cc:348] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.458359 21244 raft_consensus.cc:374] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.458380 21244 raft_consensus.cc:729] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.458362 21245 raft_consensus.cc:348] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.458338 21243 raft_consensus.cc:348] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.458447 21245 raft_consensus.cc:374] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.458482 21245 raft_consensus.cc:729] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.458487 21243 raft_consensus.cc:374] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.458479 21244 consensus_queue.cc:260] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.458526 21243 raft_consensus.cc:729] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.458609 21245 consensus_queue.cc:260] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.458636 21243 consensus_queue.cc:260] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.458977 21245 ts_tablet_manager.cc:1398] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.459076 21244 ts_tablet_manager.cc:1398] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.459070 21243 ts_tablet_manager.cc:1398] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.519709 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:58.526293 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 02470b6e208d45f3946a025b0db71e21 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.526314 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 02470b6e208d45f3946a025b0db71e21 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.526620 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02470b6e208d45f3946a025b0db71e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.526626 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02470b6e208d45f3946a025b0db71e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.529124 21249 tablet_bootstrap.cc:492] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:58.529536 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 02470b6e208d45f3946a025b0db71e21 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.529827 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02470b6e208d45f3946a025b0db71e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.530246 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 95e2524a718d4ed993027a589e740e47 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.530246 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 95e2524a718d4ed993027a589e740e47 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.530457 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e2524a718d4ed993027a589e740e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.530555 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 95e2524a718d4ed993027a589e740e47 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.531047 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 9057acb68847420fac79c901a8d89fe3 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.531116 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 9057acb68847420fac79c901a8d89fe3 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.531312 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9057acb68847420fac79c901a8d89fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.531774 14936 tablet_service.cc:1467] Processing CreateTablet for tablet 9057acb68847420fac79c901a8d89fe3 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4e93d905029542a18af13da9542786d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:58.532061 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9057acb68847420fac79c901a8d89fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.532334 21250 tablet_bootstrap.cc:492] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:58.533217 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9057acb68847420fac79c901a8d89fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.533270 21250 tablet_bootstrap.cc:654] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.533756 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e2524a718d4ed993027a589e740e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.534376 21243 tablet_bootstrap.cc:492] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:58.534595 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e2524a718d4ed993027a589e740e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:58.535482 21243 tablet_bootstrap.cc:654] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.536296 21244 tablet_bootstrap.cc:492] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:58.536688 21243 tablet_bootstrap.cc:492] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:58.536765 21243 ts_tablet_manager.cc:1367] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:09:58.537245 21243 raft_consensus.cc:348] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.537358 21243 raft_consensus.cc:374] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.537405 21243 raft_consensus.cc:729] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.537416 21244 tablet_bootstrap.cc:654] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.537510 21243 consensus_queue.cc:260] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.537835 21243 ts_tablet_manager.cc:1398] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.538395 21249 tablet_bootstrap.cc:654] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.539453 21251 tablet_bootstrap.cc:492] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:58.540671 21253 tablet_bootstrap.cc:492] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:58.540870 21251 tablet_bootstrap.cc:654] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.541134 21250 tablet_bootstrap.cc:492] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:58.541224 21250 ts_tablet_manager.cc:1367] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:09:58.541667 21250 raft_consensus.cc:348] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.541779 21250 raft_consensus.cc:374] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.541811 21250 raft_consensus.cc:729] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.541996 21253 tablet_bootstrap.cc:654] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.541940 21250 consensus_queue.cc:260] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.542244 21250 ts_tablet_manager.cc:1398] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.542388 21250 tablet_bootstrap.cc:492] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:58.542915 21251 tablet_bootstrap.cc:492] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:58.542986 21251 ts_tablet_manager.cc:1367] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:58.543246 21253 tablet_bootstrap.cc:492] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:58.543324 21250 tablet_bootstrap.cc:654] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.543327 21253 ts_tablet_manager.cc:1367] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:58.543321 21251 raft_consensus.cc:348] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.543396 21251 raft_consensus.cc:374] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.543416 21251 raft_consensus.cc:729] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.543514 21251 consensus_queue.cc:260] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.543737 21251 ts_tablet_manager.cc:1398] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:58.543773 21253 raft_consensus.cc:348] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.543874 21253 raft_consensus.cc:374] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.543903 21253 raft_consensus.cc:729] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.544070 21253 consensus_queue.cc:260] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.545950 21250 tablet_bootstrap.cc:492] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:58.545940 21253 ts_tablet_manager.cc:1398] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:09:58.546082 21250 ts_tablet_manager.cc:1367] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:09:58.546571 21250 raft_consensus.cc:348] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.546757 21250 raft_consensus.cc:374] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.546838 21250 raft_consensus.cc:729] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.546970 21249 tablet_bootstrap.cc:492] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:58.546994 21250 consensus_queue.cc:260] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.547073 21249 ts_tablet_manager.cc:1367] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:09:58.547258 21250 ts_tablet_manager.cc:1398] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:09:58.547516 21249 raft_consensus.cc:348] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.547624 21249 raft_consensus.cc:374] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.547654 21249 raft_consensus.cc:729] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.547765 21249 consensus_queue.cc:260] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.548086 21249 ts_tablet_manager.cc:1398] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.548252 21249 tablet_bootstrap.cc:492] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:58.548538 21244 tablet_bootstrap.cc:492] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:58.548632 21244 ts_tablet_manager.cc:1367] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240501 14:09:58.549108 21249 tablet_bootstrap.cc:654] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.549072 21244 raft_consensus.cc:348] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.549176 21244 raft_consensus.cc:374] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.549206 21244 raft_consensus.cc:729] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.549337 21244 consensus_queue.cc:260] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.549636 21244 ts_tablet_manager.cc:1398] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:09:58.550326 21174 raft_consensus.cc:482] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.550416 21174 raft_consensus.cc:504] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.550859 21174 leader_election.cc:288] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:58.551097 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02470b6e208d45f3946a025b0db71e21" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:58.551196 12978 raft_consensus.cc:2454] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:58.551951 12522 leader_election.cc:302] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:58.552603 21174 raft_consensus.cc:2781] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.552660 21174 raft_consensus.cc:482] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.552697 21174 raft_consensus.cc:3037] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.552714 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02470b6e208d45f3946a025b0db71e21" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:58.552822 12717 raft_consensus.cc:2454] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:09:58.553778 21174 raft_consensus.cc:504] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.554091 21174 leader_election.cc:288] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:09:58.554796 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02470b6e208d45f3946a025b0db71e21" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:58.554836 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02470b6e208d45f3946a025b0db71e21" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:58.554912 12978 raft_consensus.cc:3037] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.554948 12717 raft_consensus.cc:3037] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.555858 12978 raft_consensus.cc:2454] T 02470b6e208d45f3946a025b0db71e21 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:58.555981 12717 raft_consensus.cc:2454] T 02470b6e208d45f3946a025b0db71e21 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:09:58.556279 12520 leader_election.cc:302] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
W20240501 14:09:58.557001 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:58.557165 21249 tablet_bootstrap.cc:492] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:58.557256 21249 ts_tablet_manager.cc:1367] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:09:58.557577 21174 raft_consensus.cc:2781] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.557658 21174 raft_consensus.cc:686] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:09:58.557751 21249 raft_consensus.cc:348] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.557785 21174 consensus_queue.cc:237] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:58.557861 21249 raft_consensus.cc:374] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.557896 21249 raft_consensus.cc:729] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.559228 21249 consensus_queue.cc:260] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.559625 21249 ts_tablet_manager.cc:1398] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:58.560125 21249 tablet_bootstrap.cc:492] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:58.561148 21249 tablet_bootstrap.cc:654] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:58.561409 12440 catalog_manager.cc:5484] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:58.563311 21249 tablet_bootstrap.cc:492] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:58.563441 21249 ts_tablet_manager.cc:1367] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:58.563897 21249 raft_consensus.cc:348] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.564066 21249 raft_consensus.cc:374] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:58.564147 21249 raft_consensus.cc:729] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:58.564330 21249 consensus_queue.cc:260] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.564886 21249 ts_tablet_manager.cc:1398] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:58.592171 20979 raft_consensus.cc:482] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.592281 20979 raft_consensus.cc:504] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.592665 20979 leader_election.cc:288] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:58.593006 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d67e0213a2e406e9e6df4413777e3f8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:58.593132 14958 raft_consensus.cc:2454] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:58.593174 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d67e0213a2e406e9e6df4413777e3f8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:58.593325 12849 raft_consensus.cc:2454] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:58.593375 12652 leader_election.cc:302] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:58.593509 20979 raft_consensus.cc:2781] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.593542 20979 raft_consensus.cc:482] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.593575 20979 raft_consensus.cc:3037] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.594573 20979 raft_consensus.cc:504] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.594959 20979 leader_election.cc:288] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:58.595191 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d67e0213a2e406e9e6df4413777e3f8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:58.595249 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d67e0213a2e406e9e6df4413777e3f8" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:58.595290 14958 raft_consensus.cc:3037] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.595364 12849 raft_consensus.cc:3037] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.596132 12849 raft_consensus.cc:2454] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:58.596200 14958 raft_consensus.cc:2454] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:58.596488 12652 leader_election.cc:302] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:58.596712 20979 raft_consensus.cc:2781] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.596783 20979 raft_consensus.cc:686] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:58.596925 20979 consensus_queue.cc:237] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:58.599340 12440 catalog_manager.cc:5484] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:58.608933 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:58.674265 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:58.681766 21056 raft_consensus.cc:482] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.681879 21056 raft_consensus.cc:504] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.682247 21056 leader_election.cc:288] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:58.682569 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e2524a718d4ed993027a589e740e47" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:58.682628 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e2524a718d4ed993027a589e740e47" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:58.682701 12978 raft_consensus.cc:2454] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:58.682799 12585 raft_consensus.cc:2454] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:58.683030 15156 leader_election.cc:302] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:58.683202 21056 raft_consensus.cc:2781] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.683259 21056 raft_consensus.cc:482] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.683313 21056 raft_consensus.cc:3037] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.684356 21056 raft_consensus.cc:504] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.684689 21056 leader_election.cc:288] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:09:58.684994 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e2524a718d4ed993027a589e740e47" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:58.684994 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e2524a718d4ed993027a589e740e47" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:58.685110 12978 raft_consensus.cc:3037] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.685120 12585 raft_consensus.cc:3037] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.686105 12978 raft_consensus.cc:2454] T 95e2524a718d4ed993027a589e740e47 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:58.686110 12585 raft_consensus.cc:2454] T 95e2524a718d4ed993027a589e740e47 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:58.686419 15156 leader_election.cc:302] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:09:58.686599 21056 raft_consensus.cc:2781] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.686686 21056 raft_consensus.cc:686] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:58.686839 21056 consensus_queue.cc:237] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:58.689067 12440 catalog_manager.cc:5484] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:58.717214 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:58.721602 14958 raft_consensus.cc:1261] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:58.722221 21157 consensus_queue.cc:1035] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.722533 12848 raft_consensus.cc:1261] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:58.724123 21157 consensus_queue.cc:1035] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.725332 12440 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet 0d67e0213a2e406e9e6df4413777e3f8 (table rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792]). Expected version 1 got 0
I20240501 14:09:58.727068 21277 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.727313 21205 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.727428 21277 tablet.cc:1620] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:58.727604 21205 tablet.cc:1620] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:58.728379 21208 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:58.728683 21208 tablet.cc:1620] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:58.729275 12440 catalog_manager.cc:5576] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215) does not have the latest schema for tablet 0d67e0213a2e406e9e6df4413777e3f8 (table rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792]). Expected version 1 got 0
I20240501 14:09:58.732199 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792] alter complete (version 1)
W20240501 14:09:58.753394 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:58.812790 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 14:09:58.813505 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:58.835934 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:58.841444 21277 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.841615 21208 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.841806 21277 tablet.cc:1620] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:58.841879 21208 tablet.cc:1620] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:09:58.843564 21205 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:58.843866 21205 tablet.cc:1620] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:09:58.847824 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=5340d6f66f4d441089db38fd023f1792] alter complete (version 2)
I20240501 14:09:58.915588 21172 raft_consensus.cc:482] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:58.915724 21172 raft_consensus.cc:504] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.916126 21172 leader_election.cc:288] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:58.916484 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9057acb68847420fac79c901a8d89fe3" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:58.916626 15352 raft_consensus.cc:2454] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:58.916707 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9057acb68847420fac79c901a8d89fe3" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:58.916857 14958 raft_consensus.cc:2454] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:09:58.916923 12916 leader_election.cc:302] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:58.917114 21172 raft_consensus.cc:2781] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:58.917181 21172 raft_consensus.cc:482] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:58.917234 21172 raft_consensus.cc:3037] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.918081 21172 raft_consensus.cc:504] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.918437 21172 leader_election.cc:288] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:58.918815 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9057acb68847420fac79c901a8d89fe3" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:58.918947 15352 raft_consensus.cc:3037] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.919063 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9057acb68847420fac79c901a8d89fe3" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:58.919175 14958 raft_consensus.cc:3037] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:58.919929 15352 raft_consensus.cc:2454] T 9057acb68847420fac79c901a8d89fe3 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:58.920032 14958 raft_consensus.cc:2454] T 9057acb68847420fac79c901a8d89fe3 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:09:58.920205 12916 leader_election.cc:302] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:09:58.920384 21172 raft_consensus.cc:2781] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:58.920562 21172 raft_consensus.cc:686] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:09:58.920713 21172 consensus_queue.cc:237] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:58.923190 12443 catalog_manager.cc:5484] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:58.941922 21038 consensus_queue.cc:1035] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.945168 21038 consensus_queue.cc:1035] T 02470b6e208d45f3946a025b0db71e21 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:58.950765 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:58.956007 21277 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.956549 21208 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.958691 21205 tablet.cc:1722] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:58.961860 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=5340d6f66f4d441089db38fd023f1792] alter complete (version 3)
W20240501 14:09:58.992386 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:09:59.011711 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:59.065114 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:59.065330 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:59.067705 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 0d67e0213a2e406e9e6df4413777e3f8
I20240501 14:09:59.068483 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d67e0213a2e406e9e6df4413777e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:09:59.068591 12696 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d67e0213a2e406e9e6df4413777e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:59.068809 21308 tablet_replica.cc:330] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:09:59.068984 21309 tablet_replica.cc:330] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:59.068986 21308 raft_consensus.cc:2229] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.069103 21308 raft_consensus.cc:2258] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.069161 21309 raft_consensus.cc:2229] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:59.069463 21309 raft_consensus.cc:2258] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.069721 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d67e0213a2e406e9e6df4413777e3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:09:59.070003 21310 tablet_replica.cc:330] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:09:59.070154 21308 ts_tablet_manager.cc:1875] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.070185 21310 raft_consensus.cc:2229] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.070276 21309 ts_tablet_manager.cc:1875] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.070348 21310 raft_consensus.cc:2258] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.071043 21310 ts_tablet_manager.cc:1875] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.072485 21309 ts_tablet_manager.cc:1888] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.072557 21309 log.cc:1196] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/0d67e0213a2e406e9e6df4413777e3f8
I20240501 14:09:59.072592 21308 ts_tablet_manager.cc:1888] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.072708 21308 log.cc:1196] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/0d67e0213a2e406e9e6df4413777e3f8
I20240501 14:09:59.072944 21309 ts_tablet_manager.cc:1909] T 0d67e0213a2e406e9e6df4413777e3f8 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:59.072989 21308 ts_tablet_manager.cc:1909] T 0d67e0213a2e406e9e6df4413777e3f8 P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:09:59.073282 21310 ts_tablet_manager.cc:1888] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.073379 21310 log.cc:1196] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/0d67e0213a2e406e9e6df4413777e3f8
I20240501 14:09:59.073652 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 0d67e0213a2e406e9e6df4413777e3f8 (table rebalancer_dynamic_table_new_name [id=5340d6f66f4d441089db38fd023f1792]) successfully deleted
I20240501 14:09:59.073750 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet 0d67e0213a2e406e9e6df4413777e3f8 (table rebalancer_dynamic_table_new_name [id=5340d6f66f4d441089db38fd023f1792]) successfully deleted
I20240501 14:09:59.074195 21310 ts_tablet_manager.cc:1909] T 0d67e0213a2e406e9e6df4413777e3f8 P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:09:59.075670 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet 0d67e0213a2e406e9e6df4413777e3f8 (table rebalancer_dynamic_table_new_name [id=5340d6f66f4d441089db38fd023f1792]) successfully deleted
I20240501 14:09:59.113577 21056 consensus_queue.cc:1035] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.117542 20152 consensus_queue.cc:1035] T 95e2524a718d4ed993027a589e740e47 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.169710 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:59.176322 15201 tablet_service.cc:1467] Processing CreateTablet for tablet afe12d15f353459997ba597c30443631 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]), partition=
I20240501 14:09:59.176322 12696 tablet_service.cc:1467] Processing CreateTablet for tablet afe12d15f353459997ba597c30443631 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]), partition=
I20240501 14:09:59.176345 12958 tablet_service.cc:1467] Processing CreateTablet for tablet afe12d15f353459997ba597c30443631 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]), partition=
I20240501 14:09:59.176594 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe12d15f353459997ba597c30443631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.176594 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe12d15f353459997ba597c30443631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.176610 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe12d15f353459997ba597c30443631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.179199 21316 tablet_bootstrap.cc:492] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:09:59.179639 21314 tablet_bootstrap.cc:492] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:09:59.180418 21316 tablet_bootstrap.cc:654] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.180716 21314 tablet_bootstrap.cc:654] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.181048 21315 tablet_bootstrap.cc:492] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:59.181679 21314 tablet_bootstrap.cc:492] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:09:59.181816 21314 ts_tablet_manager.cc:1367] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:59.182322 21314 raft_consensus.cc:348] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.182410 21316 tablet_bootstrap.cc:492] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:09:59.182493 21316 ts_tablet_manager.cc:1367] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:59.182634 21314 raft_consensus.cc:374] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.182744 21315 tablet_bootstrap.cc:654] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.182935 21314 raft_consensus.cc:729] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.183118 21314 consensus_queue.cc:260] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.183459 21314 ts_tablet_manager.cc:1398] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:59.183684 21316 raft_consensus.cc:348] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.183779 21316 raft_consensus.cc:374] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.183815 21316 raft_consensus.cc:729] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.183977 21316 consensus_queue.cc:260] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.184329 21316 ts_tablet_manager.cc:1398] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.184376 21315 tablet_bootstrap.cc:492] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:59.184504 21315 ts_tablet_manager.cc:1367] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:59.184978 21315 raft_consensus.cc:348] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.185127 21315 raft_consensus.cc:374] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.185191 21315 raft_consensus.cc:729] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.185326 21315 consensus_queue.cc:260] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.186394 21315 ts_tablet_manager.cc:1398] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:59.236857 20152 raft_consensus.cc:482] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.236984 20152 raft_consensus.cc:504] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.237365 20152 leader_election.cc:288] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:59.237757 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe12d15f353459997ba597c30443631" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:09:59.237790 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe12d15f353459997ba597c30443631" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:09:59.237892 12717 raft_consensus.cc:2454] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:59.237929 12978 raft_consensus.cc:2454] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:09:59.238396 15154 leader_election.cc:302] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:59.238569 20152 raft_consensus.cc:2781] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.238628 20152 raft_consensus.cc:482] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.238688 20152 raft_consensus.cc:3037] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.239774 20152 raft_consensus.cc:504] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.240087 20152 leader_election.cc:288] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:09:59.240793 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe12d15f353459997ba597c30443631" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:09:59.240870 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe12d15f353459997ba597c30443631" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:09:59.240914 12717 raft_consensus.cc:3037] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.240988 12978 raft_consensus.cc:3037] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.241773 12717 raft_consensus.cc:2454] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:59.241984 12978 raft_consensus.cc:2454] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:09:59.242064 15154 leader_election.cc:302] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:59.242362 21000 raft_consensus.cc:2781] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.242460 21000 raft_consensus.cc:686] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:09:59.242590 21000 consensus_queue.cc:237] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:09:59.245697 12440 catalog_manager.cc:5484] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:09:59.328321 21292 consensus_queue.cc:1035] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.333062 21172 consensus_queue.cc:1035] T 9057acb68847420fac79c901a8d89fe3 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:09:59.345969 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:59.361616 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:09:59.366063 12978 raft_consensus.cc:1261] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:59.366629 20152 consensus_queue.cc:1035] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.367461 12717 raft_consensus.cc:1261] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:09:59.368005 21000 consensus_queue.cc:1035] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:09:59.371307 12438 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]). Expected version 1 got 0
I20240501 14:09:59.372081 12440 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]). Expected version 1 got 0
I20240501 14:09:59.373485 21338 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.373821 21338 tablet.cc:1620] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:59.374073 21277 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.374186 12440 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]). Expected version 1 got 0
I20240501 14:09:59.374379 21277 tablet.cc:1620] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:59.375902 12438 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]). Expected version 1 got 0
I20240501 14:09:59.377905 12443 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]). Expected version 1 got 0
I20240501 14:09:59.379135 12438 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9]). Expected version 1 got 0
I20240501 14:09:59.381918 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9] alter complete (version 1)
I20240501 14:09:59.382570 21337 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:09:59.382899 21337 tablet.cc:1620] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
W20240501 14:09:59.414145 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:59.481078 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:59.485443 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:09:59.491442 21337 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.491809 21337 tablet.cc:1620] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:09:59.492610 21277 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.492691 21338 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:09:59.492888 21277 tablet.cc:1620] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:09:59.492977 21338 tablet.cc:1620] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:09:59.495429 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:09:59.499203 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=4397249f3a0e467fb5edea23e9d4b8b9] alter complete (version 2)
I20240501 14:09:59.502357 12566 tablet_service.cc:1467] Processing CreateTablet for tablet add316bbed1c4f5a9bcf147337dfa904 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.502671 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add316bbed1c4f5a9bcf147337dfa904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.502628 12696 tablet_service.cc:1467] Processing CreateTablet for tablet add316bbed1c4f5a9bcf147337dfa904 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.502924 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add316bbed1c4f5a9bcf147337dfa904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.503258 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 295e27b09f8643f3ac36e305c7bfc6cb (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.503410 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 295e27b09f8643f3ac36e305c7bfc6cb (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.503504 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295e27b09f8643f3ac36e305c7bfc6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.503623 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295e27b09f8643f3ac36e305c7bfc6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.505141 21315 tablet_bootstrap.cc:492] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:59.505172 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 9632ad35118549209b23dfb25713140d (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.505403 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9632ad35118549209b23dfb25713140d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.506122 21315 tablet_bootstrap.cc:654] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.506490 14936 tablet_service.cc:1467] Processing CreateTablet for tablet add316bbed1c4f5a9bcf147337dfa904 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.506570 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 9632ad35118549209b23dfb25713140d (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.506795 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add316bbed1c4f5a9bcf147337dfa904. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.507014 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 295e27b09f8643f3ac36e305c7bfc6cb (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.507236 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 295e27b09f8643f3ac36e305c7bfc6cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.507588 21315 tablet_bootstrap.cc:492] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:59.507659 21315 ts_tablet_manager.cc:1367] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:59.508064 21315 raft_consensus.cc:348] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.508159 21315 raft_consensus.cc:374] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.508193 21315 raft_consensus.cc:729] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.508337 21315 consensus_queue.cc:260] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.508651 21315 ts_tablet_manager.cc:1398] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:59.508836 21315 tablet_bootstrap.cc:492] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:09:59.509030 21352 tablet_bootstrap.cc:492] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:59.509202 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9632ad35118549209b23dfb25713140d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.509797 21315 tablet_bootstrap.cc:654] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.509853 21352 tablet_bootstrap.cc:654] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.510859 21352 tablet_bootstrap.cc:492] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:59.510998 21352 ts_tablet_manager.cc:1367] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:09:59.511499 21352 raft_consensus.cc:348] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.511759 21352 raft_consensus.cc:374] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.511848 21352 raft_consensus.cc:729] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.512034 21352 consensus_queue.cc:260] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.512383 21352 ts_tablet_manager.cc:1398] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.512619 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 9632ad35118549209b23dfb25713140d (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=3e98c50065ae47f6934080f682b541b8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:09:59.512899 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9632ad35118549209b23dfb25713140d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.515470 21354 tablet_bootstrap.cc:492] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:59.516736 21315 tablet_bootstrap.cc:492] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:09:59.516875 21315 ts_tablet_manager.cc:1367] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:09:59.516880 21351 tablet_bootstrap.cc:492] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:59.517395 21315 raft_consensus.cc:348] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.517572 21315 raft_consensus.cc:374] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.517652 21315 raft_consensus.cc:729] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.517835 21351 tablet_bootstrap.cc:654] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.517851 21315 consensus_queue.cc:260] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.518067 21357 tablet_bootstrap.cc:492] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:59.518236 21315 ts_tablet_manager.cc:1398] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.519138 21357 tablet_bootstrap.cc:654] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.516624 21354 tablet_bootstrap.cc:654] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.520781 21357 tablet_bootstrap.cc:492] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:59.520865 21357 ts_tablet_manager.cc:1367] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:59.521060 21354 tablet_bootstrap.cc:492] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:59.521145 21354 ts_tablet_manager.cc:1367] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:09:59.521314 21357 raft_consensus.cc:348] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.521435 21357 raft_consensus.cc:374] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.521469 21357 raft_consensus.cc:729] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.521644 21354 raft_consensus.cc:348] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.521657 21357 consensus_queue.cc:260] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.521797 21354 raft_consensus.cc:374] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.522012 21357 ts_tablet_manager.cc:1398] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.521859 21354 raft_consensus.cc:729] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.523278 21354 consensus_queue.cc:260] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.523382 21351 tablet_bootstrap.cc:492] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:59.523520 21351 ts_tablet_manager.cc:1367] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:09:59.523996 21351 raft_consensus.cc:348] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.524188 21351 raft_consensus.cc:374] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.524250 21351 raft_consensus.cc:729] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.524400 21351 consensus_queue.cc:260] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.524746 21351 ts_tablet_manager.cc:1398] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.525013 21351 tablet_bootstrap.cc:492] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:09:59.525681 21354 ts_tablet_manager.cc:1398] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:09:59.525902 21354 tablet_bootstrap.cc:492] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:59.526975 21354 tablet_bootstrap.cc:654] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.527472 21351 tablet_bootstrap.cc:654] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.528548 21351 tablet_bootstrap.cc:492] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:09:59.528671 21351 ts_tablet_manager.cc:1367] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:59.529196 21354 tablet_bootstrap.cc:492] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:59.529273 21354 ts_tablet_manager.cc:1367] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:09:59.529472 21351 raft_consensus.cc:348] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.529579 21351 raft_consensus.cc:374] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.529623 21351 raft_consensus.cc:729] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.529739 21354 raft_consensus.cc:348] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.529801 21351 consensus_queue.cc:260] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.529841 21354 raft_consensus.cc:374] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.529876 21354 raft_consensus.cc:729] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.529992 21354 consensus_queue.cc:260] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.530170 21351 ts_tablet_manager.cc:1398] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.530293 21354 ts_tablet_manager.cc:1398] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.530458 21354 tablet_bootstrap.cc:492] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:59.531239 21354 tablet_bootstrap.cc:654] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.532202 21354 tablet_bootstrap.cc:492] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:59.532284 21354 ts_tablet_manager.cc:1367] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:59.532706 21354 raft_consensus.cc:348] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.532801 21354 raft_consensus.cc:374] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.532841 21354 raft_consensus.cc:729] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.532961 21354 consensus_queue.cc:260] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.533296 21354 ts_tablet_manager.cc:1398] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:09:59.559358 20777 raft_consensus.cc:482] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.559474 20777 raft_consensus.cc:504] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.559918 20777 leader_election.cc:288] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:59.560317 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295e27b09f8643f3ac36e305c7bfc6cb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:59.560458 12585 raft_consensus.cc:2454] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:59.560782 14893 leader_election.cc:302] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:59.560829 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295e27b09f8643f3ac36e305c7bfc6cb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:09:59.560979 12848 raft_consensus.cc:2454] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:09:59.561065 21364 raft_consensus.cc:2781] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.561123 21364 raft_consensus.cc:482] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.561169 21364 raft_consensus.cc:3037] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.562192 21364 raft_consensus.cc:504] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.562506 21364 leader_election.cc:288] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:09:59.562968 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295e27b09f8643f3ac36e305c7bfc6cb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:59.563108 12585 raft_consensus.cc:3037] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.564126 12585 raft_consensus.cc:2454] T 295e27b09f8643f3ac36e305c7bfc6cb P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:59.564343 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "295e27b09f8643f3ac36e305c7bfc6cb" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:09:59.564452 12848 raft_consensus.cc:3037] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.564544 14893 leader_election.cc:302] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:59.564728 21364 raft_consensus.cc:2781] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
W20240501 14:09:59.564859 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:09:59.564894 21364 raft_consensus.cc:686] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:09:59.565044 21364 consensus_queue.cc:237] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:09:59.565457 12848 raft_consensus.cc:2454] T 295e27b09f8643f3ac36e305c7bfc6cb P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:09:59.568087 12443 catalog_manager.cc:5484] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
W20240501 14:09:59.570119 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:59.603394 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:09:59.609748 21337 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:59.610436 21338 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:59.612795 21277 tablet.cc:1722] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:09:59.615792 21350 raft_consensus.cc:482] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.615898 21350 raft_consensus.cc:504] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.616339 21350 leader_election.cc:288] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:59.617539 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add316bbed1c4f5a9bcf147337dfa904" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:09:59.617686 12584 raft_consensus.cc:2454] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:59.618865 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=4397249f3a0e467fb5edea23e9d4b8b9] alter complete (version 3)
I20240501 14:09:59.619576 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add316bbed1c4f5a9bcf147337dfa904" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:59.619732 14958 raft_consensus.cc:2454] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:59.620081 12652 leader_election.cc:302] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:59.620589 21350 raft_consensus.cc:2781] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.620654 21350 raft_consensus.cc:482] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.620692 21350 raft_consensus.cc:3037] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.621774 21350 raft_consensus.cc:504] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.622164 21350 leader_election.cc:288] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:09:59.622536 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add316bbed1c4f5a9bcf147337dfa904" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:59.622606 12584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add316bbed1c4f5a9bcf147337dfa904" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:09:59.622664 14958 raft_consensus.cc:3037] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.622759 12584 raft_consensus.cc:3037] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.623726 12584 raft_consensus.cc:2454] T add316bbed1c4f5a9bcf147337dfa904 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:59.623726 14958 raft_consensus.cc:2454] T add316bbed1c4f5a9bcf147337dfa904 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:59.624069 12652 leader_election.cc:302] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:59.624281 21350 raft_consensus.cc:2781] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.624375 21350 raft_consensus.cc:686] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:59.624516 21350 consensus_queue.cc:237] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:09:59.637962 12443 catalog_manager.cc:5484] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } }
W20240501 14:09:59.674094 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:09:59.693315 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:59.711894 21350 raft_consensus.cc:482] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.712038 21350 raft_consensus.cc:504] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.712512 21350 leader_election.cc:288] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:59.712895 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9632ad35118549209b23dfb25713140d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:59.712996 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9632ad35118549209b23dfb25713140d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:59.713073 14958 raft_consensus.cc:2454] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:59.713128 15352 raft_consensus.cc:2454] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:09:59.713379 12652 leader_election.cc:302] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:59.713958 21350 raft_consensus.cc:2781] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.714018 21350 raft_consensus.cc:482] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.714056 21350 raft_consensus.cc:3037] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:09:59.714303 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:09:59.715051 21350 raft_consensus.cc:504] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.715420 21350 leader_election.cc:288] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:59.715708 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9632ad35118549209b23dfb25713140d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:59.715834 14958 raft_consensus.cc:3037] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.715811 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9632ad35118549209b23dfb25713140d" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:59.715950 15352 raft_consensus.cc:3037] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.716768 14958 raft_consensus.cc:2454] T 9632ad35118549209b23dfb25713140d P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:59.717080 15352 raft_consensus.cc:2454] T 9632ad35118549209b23dfb25713140d P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:09:59.717114 12652 leader_election.cc:302] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:09:59.717342 21350 raft_consensus.cc:2781] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.717442 21350 raft_consensus.cc:686] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:09:59.717618 21350 consensus_queue.cc:237] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.719893 12443 catalog_manager.cc:5484] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:09:59.722770 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:59.722889 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:09:59.724637 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet afe12d15f353459997ba597c30443631
I20240501 14:09:59.725200 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet afe12d15f353459997ba597c30443631 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:09:59.725307 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet afe12d15f353459997ba597c30443631 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:09:59.725618 21379 tablet_replica.cc:330] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:09:59.725613 21380 tablet_replica.cc:330] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:09:59.725595 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet afe12d15f353459997ba597c30443631 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:09:59 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:09:59.725775 21379 raft_consensus.cc:2229] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.725790 21380 raft_consensus.cc:2229] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:09:59.725893 21381 tablet_replica.cc:330] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:09:59.725914 21379 raft_consensus.cc:2258] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.725915 21380 raft_consensus.cc:2258] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.726318 21381 raft_consensus.cc:2229] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:09:59.726536 21379 ts_tablet_manager.cc:1875] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.726531 21380 ts_tablet_manager.cc:1875] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.726555 21381 raft_consensus.cc:2258] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:09:59.727087 21381 ts_tablet_manager.cc:1875] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:09:59.728761 21379 ts_tablet_manager.cc:1888] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.728869 21379 log.cc:1196] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/afe12d15f353459997ba597c30443631
I20240501 14:09:59.729226 21379 ts_tablet_manager.cc:1909] T afe12d15f353459997ba597c30443631 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:09:59.729583 21380 ts_tablet_manager.cc:1888] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.729629 21381 ts_tablet_manager.cc:1888] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:09:59.729674 21380 log.cc:1196] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/afe12d15f353459997ba597c30443631
I20240501 14:09:59.729735 21381 log.cc:1196] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/afe12d15f353459997ba597c30443631
I20240501 14:09:59.729996 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table_new_name [id=4397249f3a0e467fb5edea23e9d4b8b9]) successfully deleted
I20240501 14:09:59.730113 21381 ts_tablet_manager.cc:1909] T afe12d15f353459997ba597c30443631 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:09:59.730165 21380 ts_tablet_manager.cc:1909] T afe12d15f353459997ba597c30443631 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:09:59.730844 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table_new_name [id=4397249f3a0e467fb5edea23e9d4b8b9]) successfully deleted
I20240501 14:09:59.731123 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet afe12d15f353459997ba597c30443631 (table rebalancer_dynamic_table_new_name [id=4397249f3a0e467fb5edea23e9d4b8b9]) successfully deleted
W20240501 14:09:59.755728 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:59.758450 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:59.764257 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:59.794247 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:59.826370 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:09:59.832737 12827 tablet_service.cc:1467] Processing CreateTablet for tablet a3f65d7398be46a7ab58cbb13dbbe40b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e]), partition=
I20240501 14:09:59.832722 14937 tablet_service.cc:1467] Processing CreateTablet for tablet a3f65d7398be46a7ab58cbb13dbbe40b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e]), partition=
I20240501 14:09:59.833042 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f65d7398be46a7ab58cbb13dbbe40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.833063 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f65d7398be46a7ab58cbb13dbbe40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.835407 21352 tablet_bootstrap.cc:492] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:09:59.835605 15331 tablet_service.cc:1467] Processing CreateTablet for tablet a3f65d7398be46a7ab58cbb13dbbe40b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e]), partition=
I20240501 14:09:59.835863 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f65d7398be46a7ab58cbb13dbbe40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:09:59.836309 21352 tablet_bootstrap.cc:654] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.836939 21354 tablet_bootstrap.cc:492] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:09:59.837531 21352 tablet_bootstrap.cc:492] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:09:59.837610 21352 ts_tablet_manager.cc:1367] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:09:59.837813 21357 tablet_bootstrap.cc:492] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:09:59.837818 21354 tablet_bootstrap.cc:654] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.838179 21352 raft_consensus.cc:348] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.838296 21352 raft_consensus.cc:374] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.838330 21352 raft_consensus.cc:729] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.838476 21352 consensus_queue.cc:260] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.838670 21357 tablet_bootstrap.cc:654] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:09:59.839855 21352 ts_tablet_manager.cc:1398] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:09:59.840633 21354 tablet_bootstrap.cc:492] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:09:59.840721 21354 ts_tablet_manager.cc:1367] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:59.841181 21354 raft_consensus.cc:348] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.841284 21354 raft_consensus.cc:374] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.841316 21354 raft_consensus.cc:729] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.841440 21354 consensus_queue.cc:260] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.841598 21357 tablet_bootstrap.cc:492] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:09:59.841691 21357 ts_tablet_manager.cc:1367] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:09:59.842177 21354 ts_tablet_manager.cc:1398] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:09:59.842279 21357 raft_consensus.cc:348] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.842399 21357 raft_consensus.cc:374] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:09:59.842433 21357 raft_consensus.cc:729] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:09:59.842571 21357 consensus_queue.cc:260] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.842900 21357 ts_tablet_manager.cc:1398] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 14:09:59.845288 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:59.846230 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:09:59.878996 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:09:59.889034 21220 raft_consensus.cc:482] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:59.889186 21220 raft_consensus.cc:504] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.889657 21220 leader_election.cc:288] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:59.889927 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f65d7398be46a7ab58cbb13dbbe40b" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:09:59.890071 14958 raft_consensus.cc:2454] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:59.890280 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f65d7398be46a7ab58cbb13dbbe40b" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:09:59.890403 15352 raft_consensus.cc:2454] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:09:59.890415 12783 leader_election.cc:302] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:59.890673 20361 raft_consensus.cc:2781] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:09:59.890751 20361 raft_consensus.cc:482] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:09:59.890802 20361 raft_consensus.cc:3037] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.891886 20361 raft_consensus.cc:504] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.892220 20361 leader_election.cc:288] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:09:59.892547 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f65d7398be46a7ab58cbb13dbbe40b" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:09:59.892619 14958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f65d7398be46a7ab58cbb13dbbe40b" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:09:59.892666 15352 raft_consensus.cc:3037] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.892724 14958 raft_consensus.cc:3037] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:09:59.893431 14958 raft_consensus.cc:2454] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:59.893435 15352 raft_consensus.cc:2454] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:09:59.893728 12783 leader_election.cc:302] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:09:59.894140 20361 raft_consensus.cc:2781] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:09:59.894234 20361 raft_consensus.cc:686] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:09:59.894385 20361 consensus_queue.cc:237] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } }
I20240501 14:09:59.896783 12443 catalog_manager.cc:5484] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:00.001605 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240501 14:10:00.002801 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.007279 14958 raft_consensus.cc:1261] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:00.007342 15352 raft_consensus.cc:1261] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:00.007732 20361 consensus_queue.cc:1035] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.008061 20361 consensus_queue.cc:1035] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.011250 21403 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.011677 21399 mvcc.cc:205] Tried to move back new op lower bound from 7022889369625317376 to 7022889369168941056. Current Snapshot: MvccSnapshot[applied={T|T < 7022889369625317376}]
I20240501 14:10:00.012341 21403 tablet.cc:1620] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:10:00.012332 21405 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.012753 21405 tablet.cc:1620] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:10:00.014156 12443 catalog_manager.cc:5576] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177) does not have the latest schema for tablet a3f65d7398be46a7ab58cbb13dbbe40b (table rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e]). Expected version 1 got 0
I20240501 14:10:00.014840 21404 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.015187 21404 tablet.cc:1620] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:10:00.019084 12439 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e] alter complete (version 1)
I20240501 14:10:00.019197 12438 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet a3f65d7398be46a7ab58cbb13dbbe40b (table rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e]). Expected version 1 got 0
W20240501 14:10:00.026525 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.040318 20777 consensus_queue.cc:1035] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.044454 21364 consensus_queue.cc:1035] T 295e27b09f8643f3ac36e305c7bfc6cb P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.088786 21278 consensus_queue.cc:1035] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.093639 21157 consensus_queue.cc:1035] T add316bbed1c4f5a9bcf147337dfa904 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:10:00.097919 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:10:00.122635 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:10:00.128366 21403 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.128751 21403 tablet.cc:1620] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: MemRowSet was empty: no flush needed.
I20240501 14:10:00.130326 21404 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.130678 21404 tablet.cc:1620] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: MemRowSet was empty: no flush needed.
I20240501 14:10:00.132005 21405 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.132282 21405 tablet.cc:1620] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:10:00.132304 21157 consensus_queue.cc:1035] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.137943 21278 consensus_queue.cc:1035] T 9632ad35118549209b23dfb25713140d P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.139101 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=16dbc1cc609649038391e2fc05d96f6e] alter complete (version 2)
W20240501 14:10:00.203151 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.241288 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:10:00.247009 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:00.250423 21405 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.252350 21404 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.252677 21403 tablet.cc:1722] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.257766 12698 tablet_service.cc:1467] Processing CreateTablet for tablet fc095db59e8048b296a467fba626d73c (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.258105 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc095db59e8048b296a467fba626d73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.258421 12827 tablet_service.cc:1467] Processing CreateTablet for tablet fc095db59e8048b296a467fba626d73c (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.258697 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc095db59e8048b296a467fba626d73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.258795 12828 tablet_service.cc:1467] Processing CreateTablet for tablet b302aab90f7d4fe486e99f9bb51e616d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.259028 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b302aab90f7d4fe486e99f9bb51e616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.260565 12829 tablet_service.cc:1467] Processing CreateTablet for tablet e7006417e1f741479e9b18fa7a23c697 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.260814 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7006417e1f741479e9b18fa7a23c697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.261019 21352 tablet_bootstrap.cc:492] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:00.261826 21425 tablet_bootstrap.cc:492] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:10:00.262106 21352 tablet_bootstrap.cc:654] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.262850 21425 tablet_bootstrap.cc:654] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.263391 15201 tablet_service.cc:1467] Processing CreateTablet for tablet e7006417e1f741479e9b18fa7a23c697 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.263672 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7006417e1f741479e9b18fa7a23c697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.264425 12959 tablet_service.cc:1467] Processing CreateTablet for tablet b302aab90f7d4fe486e99f9bb51e616d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.264693 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b302aab90f7d4fe486e99f9bb51e616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.265836 12567 tablet_service.cc:1467] Processing CreateTablet for tablet fc095db59e8048b296a467fba626d73c (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.265987 12958 tablet_service.cc:1467] Processing CreateTablet for tablet e7006417e1f741479e9b18fa7a23c697 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.266100 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc095db59e8048b296a467fba626d73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.266223 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7006417e1f741479e9b18fa7a23c697. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.267951 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=16dbc1cc609649038391e2fc05d96f6e] alter complete (version 3)
I20240501 14:10:00.268852 21430 tablet_bootstrap.cc:492] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:10:00.269593 21428 tablet_bootstrap.cc:492] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:00.269762 15201 tablet_service.cc:1467] Processing CreateTablet for tablet b302aab90f7d4fe486e99f9bb51e616d (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=877e0d05c7e347d18310b31194c1fad5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.270064 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b302aab90f7d4fe486e99f9bb51e616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.270157 21430 tablet_bootstrap.cc:654] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.270681 21428 tablet_bootstrap.cc:654] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.271355 21430 tablet_bootstrap.cc:492] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:10:00.271461 21430 ts_tablet_manager.cc:1367] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:10:00.271826 21352 tablet_bootstrap.cc:492] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:00.271934 21352 ts_tablet_manager.cc:1367] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:10:00.272015 21430 raft_consensus.cc:348] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.272142 21430 raft_consensus.cc:374] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.272182 21430 raft_consensus.cc:729] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.272318 21430 consensus_queue.cc:260] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.272429 21352 raft_consensus.cc:348] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.272554 21352 raft_consensus.cc:374] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.272593 21352 raft_consensus.cc:729] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.272665 21430 ts_tablet_manager.cc:1398] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:00.272732 21352 consensus_queue.cc:260] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.273101 21352 ts_tablet_manager.cc:1398] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.273277 21352 tablet_bootstrap.cc:492] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:00.273885 21425 tablet_bootstrap.cc:492] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:10:00.274009 21425 ts_tablet_manager.cc:1367] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 14:10:00.274236 21352 tablet_bootstrap.cc:654] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.274724 21425 raft_consensus.cc:348] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.274873 21425 raft_consensus.cc:374] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.274912 21425 raft_consensus.cc:729] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.275043 21425 consensus_queue.cc:260] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.275332 21425 ts_tablet_manager.cc:1398] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.277004 21352 tablet_bootstrap.cc:492] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:00.277091 21352 ts_tablet_manager.cc:1367] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:00.277599 21352 raft_consensus.cc:348] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.277716 21352 raft_consensus.cc:374] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.277750 21352 raft_consensus.cc:729] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.277871 21352 consensus_queue.cc:260] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.278144 21352 ts_tablet_manager.cc:1398] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.278296 21352 tablet_bootstrap.cc:492] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:00.278632 21429 tablet_bootstrap.cc:492] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:00.279342 21352 tablet_bootstrap.cc:654] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.279721 21429 tablet_bootstrap.cc:654] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.281180 21429 tablet_bootstrap.cc:492] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
W20240501 14:10:00.281260 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.281309 21429 ts_tablet_manager.cc:1367] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:00.281852 21429 raft_consensus.cc:348] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.281996 21429 raft_consensus.cc:374] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.282052 21429 raft_consensus.cc:729] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.282209 21429 consensus_queue.cc:260] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.282763 21429 ts_tablet_manager.cc:1398] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.283180 21429 tablet_bootstrap.cc:492] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:00.284426 21429 tablet_bootstrap.cc:654] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.285673 21352 tablet_bootstrap.cc:492] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:00.285797 21352 ts_tablet_manager.cc:1367] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:10:00.286288 21352 raft_consensus.cc:348] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.286397 21352 raft_consensus.cc:374] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.286434 21352 raft_consensus.cc:729] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.286564 21352 consensus_queue.cc:260] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.286914 21352 ts_tablet_manager.cc:1398] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.287268 21428 tablet_bootstrap.cc:492] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:00.287379 21428 ts_tablet_manager.cc:1367] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 14:10:00.287886 21428 raft_consensus.cc:348] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.288013 21428 raft_consensus.cc:374] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.288053 21428 raft_consensus.cc:729] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.288179 21428 consensus_queue.cc:260] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.288491 21428 ts_tablet_manager.cc:1398] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:00.288672 21428 tablet_bootstrap.cc:492] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:00.289667 21428 tablet_bootstrap.cc:654] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.292955 21428 tablet_bootstrap.cc:492] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:00.293053 21428 ts_tablet_manager.cc:1367] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:10:00.293555 21428 raft_consensus.cc:348] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.293678 21428 raft_consensus.cc:374] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.293712 21428 raft_consensus.cc:729] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.293838 21428 consensus_queue.cc:260] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.294138 21428 ts_tablet_manager.cc:1398] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:00.294425 20361 raft_consensus.cc:482] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.294524 20361 raft_consensus.cc:504] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.294972 20361 leader_election.cc:288] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:00.295258 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7006417e1f741479e9b18fa7a23c697" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:10:00.295305 21429 tablet_bootstrap.cc:492] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:00.295401 21429 ts_tablet_manager.cc:1367] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:10:00.295384 15220 raft_consensus.cc:2454] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:10:00.295574 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7006417e1f741479e9b18fa7a23c697" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:10:00.295706 12782 leader_election.cc:302] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:10:00.295878 21429 raft_consensus.cc:348] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.295995 21429 raft_consensus.cc:374] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.296043 21429 raft_consensus.cc:729] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.296187 21429 consensus_queue.cc:260] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
W20240501 14:10:00.296428 12784 leader_election.cc:341] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:10:00.296486 21429 ts_tablet_manager.cc:1398] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:10:00.296768 21220 raft_consensus.cc:2781] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.296829 21220 raft_consensus.cc:482] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.296872 21220 raft_consensus.cc:3037] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.297868 21220 raft_consensus.cc:504] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.298240 21220 leader_election.cc:288] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:00.298991 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7006417e1f741479e9b18fa7a23c697" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:10:00.299119 12978 raft_consensus.cc:3037] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.300122 12978 raft_consensus.cc:2454] T e7006417e1f741479e9b18fa7a23c697 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:10:00.300415 12784 leader_election.cc:302] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:00.301848 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7006417e1f741479e9b18fa7a23c697" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:10:00.301973 15220 raft_consensus.cc:3037] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.302102 21397 raft_consensus.cc:482] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.302222 21397 raft_consensus.cc:504] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.302711 21397 leader_election.cc:288] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:10:00.303128 15220 raft_consensus.cc:2454] T e7006417e1f741479e9b18fa7a23c697 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:10:00.303174 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc095db59e8048b296a467fba626d73c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:10:00.303311 12717 raft_consensus.cc:2454] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:10:00.303952 12520 leader_election.cc:302] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:00.304122 21397 raft_consensus.cc:2781] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.304183 21397 raft_consensus.cc:482] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.304225 21397 raft_consensus.cc:3037] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.304392 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc095db59e8048b296a467fba626d73c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:10:00.304518 12848 raft_consensus.cc:2454] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:10:00.304800 21220 raft_consensus.cc:2781] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.304915 21220 raft_consensus.cc:686] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:10:00.305052 21220 consensus_queue.cc:237] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.305331 21397 raft_consensus.cc:504] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.305704 21397 leader_election.cc:288] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:10:00.306134 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc095db59e8048b296a467fba626d73c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:10:00.306252 12717 raft_consensus.cc:3037] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.307317 12717 raft_consensus.cc:2454] T fc095db59e8048b296a467fba626d73c P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:10:00.308007 12520 leader_election.cc:302] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:00.308315 12438 catalog_manager.cc:5484] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:00.308718 21397 raft_consensus.cc:2781] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.309074 21397 raft_consensus.cc:686] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:10:00.309226 21397 consensus_queue.cc:237] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } }
I20240501 14:10:00.309618 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc095db59e8048b296a467fba626d73c" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:10:00.309746 12848 raft_consensus.cc:3037] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
W20240501 14:10:00.310847 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:10:00.311009 12848 raft_consensus.cc:2454] T fc095db59e8048b296a467fba626d73c P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:10:00.313506 12438 catalog_manager.cc:5484] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:00.338845 21056 raft_consensus.cc:482] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.338968 21056 raft_consensus.cc:504] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.339362 21056 leader_election.cc:288] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:10:00.339701 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b302aab90f7d4fe486e99f9bb51e616d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:10:00.339864 12848 raft_consensus.cc:2454] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:10:00.339860 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b302aab90f7d4fe486e99f9bb51e616d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:10:00.340005 12978 raft_consensus.cc:2454] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:10:00.340214 15157 leader_election.cc:302] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:10:00.340433 21056 raft_consensus.cc:2781] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.340500 21056 raft_consensus.cc:482] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.340564 21056 raft_consensus.cc:3037] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.341746 21056 raft_consensus.cc:504] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.342099 21056 leader_election.cc:288] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:10:00.342839 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b302aab90f7d4fe486e99f9bb51e616d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:10:00.342994 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b302aab90f7d4fe486e99f9bb51e616d" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:10:00.343123 12978 raft_consensus.cc:3037] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.343037 12848 raft_consensus.cc:3037] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.344024 12978 raft_consensus.cc:2454] T b302aab90f7d4fe486e99f9bb51e616d P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:10:00.344108 12848 raft_consensus.cc:2454] T b302aab90f7d4fe486e99f9bb51e616d P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:10:00.344444 15157 leader_election.cc:302] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 66e2dc34330848c3b8c2dd15898d34c8; no voters:
I20240501 14:10:00.344625 21056 raft_consensus.cc:2781] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.344724 21056 raft_consensus.cc:686] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:10:00.344878 21056 consensus_queue.cc:237] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
W20240501 14:10:00.346412 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.347762 12438 catalog_manager.cc:5484] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:00.371361 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:00.371534 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:00.374788 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet a3f65d7398be46a7ab58cbb13dbbe40b
I20240501 14:10:00.376055 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f65d7398be46a7ab58cbb13dbbe40b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:00 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:10:00.376339 12827 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f65d7398be46a7ab58cbb13dbbe40b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:00 UTC) from {username='slave'} at 127.0.0.1:57284
I20240501 14:10:00.376368 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f65d7398be46a7ab58cbb13dbbe40b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:00 UTC) from {username='slave'} at 127.0.0.1:57682
I20240501 14:10:00.376405 21454 tablet_replica.cc:330] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:10:00.376569 21454 raft_consensus.cc:2229] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.376639 21456 tablet_replica.cc:330] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: stopping tablet replica
I20240501 14:10:00.376694 21454 raft_consensus.cc:2258] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.376792 21456 raft_consensus.cc:2229] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:00.376982 21455 tablet_replica.cc:330] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: stopping tablet replica
I20240501 14:10:00.377287 21455 raft_consensus.cc:2229] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:00.377332 21454 ts_tablet_manager.cc:1875] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:00.377317 21456 raft_consensus.cc:2258] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.377671 21455 raft_consensus.cc:2258] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:00.378141 21456 ts_tablet_manager.cc:1875] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:00.378381 21455 ts_tablet_manager.cc:1875] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:00.379803 21454 ts_tablet_manager.cc:1888] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:00.379948 21454 log.cc:1196] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/a3f65d7398be46a7ab58cbb13dbbe40b
I20240501 14:10:00.380303 21454 ts_tablet_manager.cc:1909] T a3f65d7398be46a7ab58cbb13dbbe40b P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:10:00.381330 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet a3f65d7398be46a7ab58cbb13dbbe40b (table rebalancer_dynamic_table_new_name [id=16dbc1cc609649038391e2fc05d96f6e]) successfully deleted
I20240501 14:10:00.381421 21455 ts_tablet_manager.cc:1888] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:00.381605 21455 log.cc:1196] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-2/wal/wals/a3f65d7398be46a7ab58cbb13dbbe40b
I20240501 14:10:00.381043 21456 ts_tablet_manager.cc:1888] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:00.382023 21455 ts_tablet_manager.cc:1909] T a3f65d7398be46a7ab58cbb13dbbe40b P 4902d2609e6d44d089590a42e70356b2: Deleting consensus metadata
I20240501 14:10:00.382035 21456 log.cc:1196] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-4/wal/wals/a3f65d7398be46a7ab58cbb13dbbe40b
I20240501 14:10:00.382341 21456 ts_tablet_manager.cc:1909] T a3f65d7398be46a7ab58cbb13dbbe40b P 8a13c028ba8a4a659b74af6c265df95e: Deleting consensus metadata
I20240501 14:10:00.383137 12432 catalog_manager.cc:4830] TS 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): tablet a3f65d7398be46a7ab58cbb13dbbe40b (table rebalancer_dynamic_table_new_name [id=16dbc1cc609649038391e2fc05d96f6e]) successfully deleted
I20240501 14:10:00.383225 12430 catalog_manager.cc:4830] TS 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): tablet a3f65d7398be46a7ab58cbb13dbbe40b (table rebalancer_dynamic_table_new_name [id=16dbc1cc609649038391e2fc05d96f6e]) successfully deleted
W20240501 14:10:00.439275 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.477658 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:10:00.483193 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 19e7636895614efc8f75ba720f85345e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e]), partition=
I20240501 14:10:00.483448 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 19e7636895614efc8f75ba720f85345e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e]), partition=
I20240501 14:10:00.483489 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e7636895614efc8f75ba720f85345e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.483671 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e7636895614efc8f75ba720f85345e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.485747 21425 tablet_bootstrap.cc:492] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:10:00.485747 21428 tablet_bootstrap.cc:492] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:00.486521 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 19e7636895614efc8f75ba720f85345e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e]), partition=
I20240501 14:10:00.486807 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e7636895614efc8f75ba720f85345e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.486824 21425 tablet_bootstrap.cc:654] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.486902 21428 tablet_bootstrap.cc:654] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.488010 21428 tablet_bootstrap.cc:492] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:00.488099 21428 ts_tablet_manager.cc:1367] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:00.488481 21425 tablet_bootstrap.cc:492] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:10:00.488557 21425 ts_tablet_manager.cc:1367] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:00.488596 21428 raft_consensus.cc:348] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.488708 21428 raft_consensus.cc:374] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.488744 21428 raft_consensus.cc:729] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.488828 21430 tablet_bootstrap.cc:492] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:10:00.488857 21428 consensus_queue.cc:260] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.489048 21425 raft_consensus.cc:348] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.489138 21425 raft_consensus.cc:374] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.489142 21428 ts_tablet_manager.cc:1398] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.489172 21425 raft_consensus.cc:729] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.489293 21425 consensus_queue.cc:260] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.489779 21430 tablet_bootstrap.cc:654] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.489946 21425 ts_tablet_manager.cc:1398] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:00.491089 21430 tablet_bootstrap.cc:492] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:10:00.491194 21430 ts_tablet_manager.cc:1367] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:10:00.491669 21430 raft_consensus.cc:348] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.491771 21430 raft_consensus.cc:374] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.491814 21430 raft_consensus.cc:729] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.491932 21430 consensus_queue.cc:260] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.493882 21430 ts_tablet_manager.cc:1398] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:10:00.523339 21452 raft_consensus.cc:482] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.523479 21452 raft_consensus.cc:504] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.523955 21452 leader_election.cc:288] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:10:00.524457 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e7636895614efc8f75ba720f85345e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:10:00.524600 12585 raft_consensus.cc:2454] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:10:00.525087 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e7636895614efc8f75ba720f85345e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:10:00.525228 12717 raft_consensus.cc:2454] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:10:00.525604 15154 leader_election.cc:302] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:00.525774 21452 raft_consensus.cc:2781] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.525833 21452 raft_consensus.cc:482] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.525873 21452 raft_consensus.cc:3037] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.526937 21452 raft_consensus.cc:504] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.527277 21452 leader_election.cc:288] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:10:00.527526 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e7636895614efc8f75ba720f85345e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:10:00.527658 12585 raft_consensus.cc:3037] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.527743 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e7636895614efc8f75ba720f85345e" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:10:00.527869 12717 raft_consensus.cc:3037] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.528502 12585 raft_consensus.cc:2454] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:10:00.528672 12717 raft_consensus.cc:2454] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:10:00.528859 15156 leader_election.cc:302] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:10:00.529049 21452 raft_consensus.cc:2781] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.529166 21452 raft_consensus.cc:686] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:10:00.529323 21452 consensus_queue.cc:237] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:00.532563 12443 catalog_manager.cc:5484] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } }
W20240501 14:10:00.598138 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.639341 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:10:00.645395 12584 raft_consensus.cc:1261] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:00.645841 21452 consensus_queue.cc:1035] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.646004 12717 raft_consensus.cc:1261] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:00.646598 21452 consensus_queue.cc:1035] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.649820 12443 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 19e7636895614efc8f75ba720f85345e (table rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e]). Expected version 1 got 0
I20240501 14:10:00.650512 21478 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.651099 21478 tablet.cc:1620] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:10:00.652853 21479 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.653264 21479 tablet.cc:1620] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:10:00.653843 21474 mvcc.cc:205] Tried to move back new op lower bound from 7022889372238319616 to 7022889371769360384. Current Snapshot: MvccSnapshot[applied={T|T < 7022889372238319616}]
I20240501 14:10:00.654134 21480 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:00.654562 21480 tablet.cc:1620] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:10:00.654601 12438 catalog_manager.cc:5576] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113) does not have the latest schema for tablet 19e7636895614efc8f75ba720f85345e (table rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e]). Expected version 1 got 0
I20240501 14:10:00.655705 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e] alter complete (version 1)
W20240501 14:10:00.681557 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:10:00.730038 21220 consensus_queue.cc:1035] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.733865 20361 consensus_queue.cc:1035] T e7006417e1f741479e9b18fa7a23c697 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.749011 21397 consensus_queue.cc:1035] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.752323 21038 consensus_queue.cc:1035] T fc095db59e8048b296a467fba626d73c P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.755103 21452 consensus_queue.cc:1035] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.758323 12438 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:10:00.759600 20152 consensus_queue.cc:1035] T b302aab90f7d4fe486e99f9bb51e616d P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:00.767973 21478 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.768335 21478 tablet.cc:1620] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:10:00.768424 21480 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.768720 21480 tablet.cc:1620] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:10:00.769763 21479 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:00.770079 21479 tablet.cc:1620] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: MemRowSet was empty: no flush needed.
I20240501 14:10:00.774876 12438 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=064f00e972614810b1fb54f485e6b01e] alter complete (version 2)
I20240501 14:10:00.853875 12439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:00.860845 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 1f72db8c465f43fb9bef6e8c42303ea7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.861203 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f72db8c465f43fb9bef6e8c42303ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.861443 12958 tablet_service.cc:1467] Processing CreateTablet for tablet d3dbe3b75f1d4c8d87e5acab12d2b515 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.861742 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3dbe3b75f1d4c8d87e5acab12d2b515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.863379 12698 tablet_service.cc:1467] Processing CreateTablet for tablet d3dbe3b75f1d4c8d87e5acab12d2b515 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.863672 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3dbe3b75f1d4c8d87e5acab12d2b515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.863924 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 9792f7aacce04968b694259014ceffbe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.864212 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9792f7aacce04968b694259014ceffbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.864315 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 9792f7aacce04968b694259014ceffbe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.864534 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9792f7aacce04968b694259014ceffbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.865720 21425 tablet_bootstrap.cc:492] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:10:00.866827 21425 tablet_bootstrap.cc:654] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.866796 12827 tablet_service.cc:1467] Processing CreateTablet for tablet 1f72db8c465f43fb9bef6e8c42303ea7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.866823 14937 tablet_service.cc:1467] Processing CreateTablet for tablet d3dbe3b75f1d4c8d87e5acab12d2b515 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.867035 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f72db8c465f43fb9bef6e8c42303ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.867061 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3dbe3b75f1d4c8d87e5acab12d2b515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.869029 21425 tablet_bootstrap.cc:492] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:10:00.869118 21425 ts_tablet_manager.cc:1367] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:10:00.869390 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 9792f7aacce04968b694259014ceffbe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.869365 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 1f72db8c465f43fb9bef6e8c42303ea7 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=846fba75e99542bb8b6d6f1374a03130]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:00.869606 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9792f7aacce04968b694259014ceffbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.869619 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f72db8c465f43fb9bef6e8c42303ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:00.869616 21425 raft_consensus.cc:348] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.869719 21425 raft_consensus.cc:374] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.869750 21425 raft_consensus.cc:729] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.869859 21425 consensus_queue.cc:260] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.870121 21425 ts_tablet_manager.cc:1398] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:00.872299 21428 tablet_bootstrap.cc:492] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:00.873317 21428 tablet_bootstrap.cc:654] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.873337 21503 tablet_bootstrap.cc:492] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:10:00.874378 21503 tablet_bootstrap.cc:654] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.875063 21428 tablet_bootstrap.cc:492] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:00.875165 21428 ts_tablet_manager.cc:1367] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:00.875182 21500 tablet_bootstrap.cc:492] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:00.875627 21428 raft_consensus.cc:348] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.875770 21428 raft_consensus.cc:374] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.875802 21428 raft_consensus.cc:729] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.875926 21428 consensus_queue.cc:260] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.876250 21500 tablet_bootstrap.cc:654] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.876282 21428 ts_tablet_manager.cc:1398] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:00.876955 12439 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:10:00.877532 21503 tablet_bootstrap.cc:492] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:10:00.878366 21503 ts_tablet_manager.cc:1367] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:00.878654 21499 tablet_bootstrap.cc:492] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:10:00.879305 21503 raft_consensus.cc:348] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.879494 21503 raft_consensus.cc:374] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.879585 21503 raft_consensus.cc:729] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.879782 21503 consensus_queue.cc:260] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.879793 21499 tablet_bootstrap.cc:654] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.880195 21503 ts_tablet_manager.cc:1398] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:00.880411 21503 tablet_bootstrap.cc:492] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:10:00.881400 21503 tablet_bootstrap.cc:654] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.885175 21501 tablet_bootstrap.cc:492] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:00.885736 21500 tablet_bootstrap.cc:492] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:00.885907 21500 ts_tablet_manager.cc:1367] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:10:00.886408 21501 tablet_bootstrap.cc:654] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.886487 21500 raft_consensus.cc:348] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.886819 21500 raft_consensus.cc:374] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.886898 21500 raft_consensus.cc:729] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.887034 21500 consensus_queue.cc:260] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.887823 21501 tablet_bootstrap.cc:492] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:00.887945 21501 ts_tablet_manager.cc:1367] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:00.888273 21500 ts_tablet_manager.cc:1398] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:00.888455 21500 tablet_bootstrap.cc:492] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:00.888464 21501 raft_consensus.cc:348] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.888584 21501 raft_consensus.cc:374] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.888628 21501 raft_consensus.cc:729] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.888752 21501 consensus_queue.cc:260] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.889438 21500 tablet_bootstrap.cc:654] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.890039 21501 ts_tablet_manager.cc:1398] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:00.890275 21501 tablet_bootstrap.cc:492] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:00.891206 21499 tablet_bootstrap.cc:492] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:10:00.891319 21499 ts_tablet_manager.cc:1367] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 14:10:00.891498 21501 tablet_bootstrap.cc:654] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:00.892066 21499 raft_consensus.cc:348] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.892237 21499 raft_consensus.cc:374] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.892269 21499 raft_consensus.cc:729] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.892385 21499 consensus_queue.cc:260] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.893828 21499 ts_tablet_manager.cc:1398] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:00.894532 21503 tablet_bootstrap.cc:492] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:10:00.894641 21503 ts_tablet_manager.cc:1367] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 14:10:00.895151 21503 raft_consensus.cc:348] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.895268 21503 raft_consensus.cc:374] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.895299 21503 raft_consensus.cc:729] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.895429 21503 consensus_queue.cc:260] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.895752 21503 ts_tablet_manager.cc:1398] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:00.897643 21473 raft_consensus.cc:482] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.897729 21501 tablet_bootstrap.cc:492] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:00.897733 21473 raft_consensus.cc:504] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.897809 21501 ts_tablet_manager.cc:1367] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:10:00.898125 21473 leader_election.cc:288] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:10:00.898279 21501 raft_consensus.cc:348] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.898391 21501 raft_consensus.cc:374] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.898420 21501 raft_consensus.cc:729] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.898519 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dbe3b75f1d4c8d87e5acab12d2b515" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:10:00.898535 21501 consensus_queue.cc:260] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.898638 14956 raft_consensus.cc:2454] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:10:00.898909 21501 ts_tablet_manager.cc:1398] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:00.898959 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dbe3b75f1d4c8d87e5acab12d2b515" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:10:00.899040 12652 leader_election.cc:302] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:00.899089 12978 raft_consensus.cc:2454] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 0.
I20240501 14:10:00.899325 21473 raft_consensus.cc:2781] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.899386 21473 raft_consensus.cc:482] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.899431 21473 raft_consensus.cc:3037] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.899541 21364 raft_consensus.cc:482] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.899632 21364 raft_consensus.cc:504] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.899967 21364 leader_election.cc:288] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:10:00.900300 21478 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.900609 21473 raft_consensus.cc:504] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.900938 21473 leader_election.cc:288] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:10:00.901154 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9792f7aacce04968b694259014ceffbe" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
W20240501 14:10:00.901531 14893 leader_election.cc:341] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:10:00.902069 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9792f7aacce04968b694259014ceffbe" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:10:00.902189 12848 raft_consensus.cc:2454] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:10:00.902662 14956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dbe3b75f1d4c8d87e5acab12d2b515" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:10:00.902812 14956 raft_consensus.cc:3037] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.903860 14956 raft_consensus.cc:2454] T d3dbe3b75f1d4c8d87e5acab12d2b515 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:10:00.904206 12652 leader_election.cc:302] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:00.904868 21278 raft_consensus.cc:2781] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.904951 21278 raft_consensus.cc:686] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Running, Role: LEADER
I20240501 14:10:00.905084 21278 consensus_queue.cc:237] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.911537 12440 catalog_manager.cc:5484] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2). New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:00.912374 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3dbe3b75f1d4c8d87e5acab12d2b515" candidate_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:10:00.912405 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=064f00e972614810b1fb54f485e6b01e] alter complete (version 3)
I20240501 14:10:00.912497 12978 raft_consensus.cc:3037] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.913630 12978 raft_consensus.cc:2454] T d3dbe3b75f1d4c8d87e5acab12d2b515 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ab42bc38694f7f9fc8bfcc0a1483b7 in term 1.
I20240501 14:10:00.915443 21480 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.920549 21479 tablet.cc:1722] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:00.921411 14894 leader_election.cc:302] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters: a6e94562bee149c2ace25672f0f9bf50
I20240501 14:10:00.921624 21364 raft_consensus.cc:2781] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.921694 21364 raft_consensus.cc:482] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.921722 21364 raft_consensus.cc:3037] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.922760 21364 raft_consensus.cc:504] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.923177 21364 leader_election.cc:288] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151)
I20240501 14:10:00.923643 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9792f7aacce04968b694259014ceffbe" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:10:00.923787 12848 raft_consensus.cc:3037] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.923967 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9792f7aacce04968b694259014ceffbe" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
W20240501 14:10:00.924367 14893 leader_election.cc:341] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:10:00.924706 21500 tablet_bootstrap.cc:492] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:00.924779 21500 ts_tablet_manager.cc:1367] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240501 14:10:00.924777 12848 raft_consensus.cc:2454] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:10:00.925238 14894 leader_election.cc:302] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e; no voters: a6e94562bee149c2ace25672f0f9bf50
I20240501 14:10:00.925282 21500 raft_consensus.cc:348] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.925384 21500 raft_consensus.cc:374] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:00.925412 21500 raft_consensus.cc:729] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:00.925530 21500 consensus_queue.cc:260] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.925601 21364 raft_consensus.cc:2781] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.925685 21364 raft_consensus.cc:686] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:10:00.925820 21364 consensus_queue.cc:237] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:00.926846 21500 ts_tablet_manager.cc:1398] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:00.928500 12440 catalog_manager.cc:5484] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:00.979367 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:00.992009 21056 raft_consensus.cc:482] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:00.992146 21056 raft_consensus.cc:504] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.992604 21056 leader_election.cc:288] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:10:00.993042 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f72db8c465f43fb9bef6e8c42303ea7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:10:00.993019 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f72db8c465f43fb9bef6e8c42303ea7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:10:00.993187 15352 raft_consensus.cc:2454] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:10:00.993187 12848 raft_consensus.cc:2454] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 0.
I20240501 14:10:00.993499 15157 leader_election.cc:302] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:10:00.993698 21056 raft_consensus.cc:2781] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:00.993770 21056 raft_consensus.cc:482] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:00.993824 21056 raft_consensus.cc:3037] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.994685 21056 raft_consensus.cc:504] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:00.995095 21056 leader_election.cc:288] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Requested vote from peers f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801), 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215)
I20240501 14:10:00.995401 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f72db8c465f43fb9bef6e8c42303ea7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:10:00.995594 15352 raft_consensus.cc:3037] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.995548 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f72db8c465f43fb9bef6e8c42303ea7" candidate_uuid: "66e2dc34330848c3b8c2dd15898d34c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:10:00.995693 12848 raft_consensus.cc:3037] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:00.996676 12848 raft_consensus.cc:2454] T 1f72db8c465f43fb9bef6e8c42303ea7 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:10:00.996676 15352 raft_consensus.cc:2454] T 1f72db8c465f43fb9bef6e8c42303ea7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66e2dc34330848c3b8c2dd15898d34c8 in term 1.
I20240501 14:10:00.996999 15157 leader_election.cc:302] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:10:00.997188 21056 raft_consensus.cc:2781] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:00.997289 21056 raft_consensus.cc:686] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Running, Role: LEADER
I20240501 14:10:00.997447 21056 consensus_queue.cc:237] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:01.002131 12438 catalog_manager.cc:5484] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7). New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:01.014387 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:01.014559 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:01.016768 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 19e7636895614efc8f75ba720f85345e
I20240501 14:10:01.017501 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e7636895614efc8f75ba720f85345e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:10:01.017536 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e7636895614efc8f75ba720f85345e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:10:01.017769 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e7636895614efc8f75ba720f85345e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:54380
I20240501 14:10:01.017853 21526 tablet_replica.cc:330] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:10:01.017876 21527 tablet_replica.cc:330] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:10:01.018000 21526 raft_consensus.cc:2229] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.018079 21527 raft_consensus.cc:2229] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:01.018162 21528 tablet_replica.cc:330] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: stopping tablet replica
I20240501 14:10:01.018162 21526 raft_consensus.cc:2258] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.018419 21527 raft_consensus.cc:2258] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.018637 21528 raft_consensus.cc:2229] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.018882 21528 raft_consensus.cc:2258] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.019331 21526 ts_tablet_manager.cc:1875] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.019363 21527 ts_tablet_manager.cc:1875] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.019520 21528 ts_tablet_manager.cc:1875] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.021585 21526 ts_tablet_manager.cc:1888] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.021698 21526 log.cc:1196] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/19e7636895614efc8f75ba720f85345e
I20240501 14:10:01.021837 21527 ts_tablet_manager.cc:1888] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.022022 21527 log.cc:1196] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/19e7636895614efc8f75ba720f85345e
I20240501 14:10:01.022184 21528 ts_tablet_manager.cc:1888] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.022310 21528 log.cc:1196] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-1/wal/wals/19e7636895614efc8f75ba720f85345e
I20240501 14:10:01.022567 21527 ts_tablet_manager.cc:1909] T 19e7636895614efc8f75ba720f85345e P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:10:01.022661 21528 ts_tablet_manager.cc:1909] T 19e7636895614efc8f75ba720f85345e P e6ab42bc38694f7f9fc8bfcc0a1483b7: Deleting consensus metadata
I20240501 14:10:01.023377 12429 catalog_manager.cc:4830] TS e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): tablet 19e7636895614efc8f75ba720f85345e (table rebalancer_dynamic_table_new_name [id=064f00e972614810b1fb54f485e6b01e]) successfully deleted
I20240501 14:10:01.023569 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 19e7636895614efc8f75ba720f85345e (table rebalancer_dynamic_table_new_name [id=064f00e972614810b1fb54f485e6b01e]) successfully deleted
I20240501 14:10:01.023667 21526 ts_tablet_manager.cc:1909] T 19e7636895614efc8f75ba720f85345e P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:10:01.024382 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 19e7636895614efc8f75ba720f85345e (table rebalancer_dynamic_table_new_name [id=064f00e972614810b1fb54f485e6b01e]) successfully deleted
I20240501 14:10:01.119050 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:10:01.124934 12567 tablet_service.cc:1467] Processing CreateTablet for tablet ffdc7e8f39c74b8887bd1e7431b19a10 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1]), partition=
I20240501 14:10:01.125146 15331 tablet_service.cc:1467] Processing CreateTablet for tablet ffdc7e8f39c74b8887bd1e7431b19a10 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1]), partition=
I20240501 14:10:01.125146 12959 tablet_service.cc:1467] Processing CreateTablet for tablet ffdc7e8f39c74b8887bd1e7431b19a10 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1]), partition=
I20240501 14:10:01.125273 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdc7e8f39c74b8887bd1e7431b19a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.125398 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdc7e8f39c74b8887bd1e7431b19a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.125398 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffdc7e8f39c74b8887bd1e7431b19a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.127602 21499 tablet_bootstrap.cc:492] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:10:01.127820 21500 tablet_bootstrap.cc:492] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:01.128026 21529 tablet_bootstrap.cc:492] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:10:01.128605 21499 tablet_bootstrap.cc:654] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.128669 21500 tablet_bootstrap.cc:654] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.128770 21529 tablet_bootstrap.cc:654] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.130071 21500 tablet_bootstrap.cc:492] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:01.130172 21500 ts_tablet_manager.cc:1367] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:01.130656 21500 raft_consensus.cc:348] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.130786 21500 raft_consensus.cc:374] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.130831 21500 raft_consensus.cc:729] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.130949 21500 consensus_queue.cc:260] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.131251 21500 ts_tablet_manager.cc:1398] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.132638 21499 tablet_bootstrap.cc:492] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:10:01.132736 21499 ts_tablet_manager.cc:1367] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:10:01.133316 21499 raft_consensus.cc:348] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.133422 21499 raft_consensus.cc:374] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.133455 21499 raft_consensus.cc:729] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.133558 21529 tablet_bootstrap.cc:492] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:10:01.133575 21499 consensus_queue.cc:260] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.133644 21529 ts_tablet_manager.cc:1367] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:10:01.133873 21499 ts_tablet_manager.cc:1398] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:01.134197 21529 raft_consensus.cc:348] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.134332 21529 raft_consensus.cc:374] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.134397 21529 raft_consensus.cc:729] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.134548 21529 consensus_queue.cc:260] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.134965 21529 ts_tablet_manager.cc:1398] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.138356 21396 raft_consensus.cc:482] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.138464 21396 raft_consensus.cc:504] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.138967 21396 leader_election.cc:288] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:10:01.139315 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdc7e8f39c74b8887bd1e7431b19a10" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:10:01.139330 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdc7e8f39c74b8887bd1e7431b19a10" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:10:01.139456 15352 raft_consensus.cc:2454] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:10:01.139456 12585 raft_consensus.cc:2454] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:10:01.139772 12916 leader_election.cc:302] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:10:01.140722 21396 raft_consensus.cc:2781] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.140792 21396 raft_consensus.cc:482] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.140836 21396 raft_consensus.cc:3037] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.141779 21396 raft_consensus.cc:504] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.142108 21396 leader_election.cc:288] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:10:01.142395 15352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdc7e8f39c74b8887bd1e7431b19a10" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:10:01.142392 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffdc7e8f39c74b8887bd1e7431b19a10" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:10:01.142508 15352 raft_consensus.cc:3037] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.142508 12585 raft_consensus.cc:3037] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.143575 12585 raft_consensus.cc:2454] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:10:01.143675 15352 raft_consensus.cc:2454] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:10:01.143932 12915 leader_election.cc:302] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:01.144408 21396 raft_consensus.cc:2781] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.144507 21396 raft_consensus.cc:686] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:10:01.144644 21396 consensus_queue.cc:237] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.147027 12440 catalog_manager.cc:5484] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } }
I20240501 14:10:01.209398 21292 raft_consensus.cc:482] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.209566 21292 raft_consensus.cc:504] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } }
I20240501 14:10:01.210004 21292 leader_election.cc:288] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:10:01.210312 12848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9792f7aacce04968b694259014ceffbe" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:10:01.210482 12848 raft_consensus.cc:2379] T 9792f7aacce04968b694259014ceffbe P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6e94562bee149c2ace25672f0f9bf50 in current term 1: Already voted for candidate 8a13c028ba8a4a659b74af6c265df95e in this term.
I20240501 14:10:01.210418 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9792f7aacce04968b694259014ceffbe" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:10:01.211040 12914 leader_election.cc:302] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e94562bee149c2ace25672f0f9bf50; no voters: 4902d2609e6d44d089590a42e70356b2, 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:10:01.211444 21292 raft_consensus.cc:3037] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.212471 21292 raft_consensus.cc:2726] T 9792f7aacce04968b694259014ceffbe P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 14:10:01.219063 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:01.232494 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:01.235690 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:01.251485 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:10:01.256278 15351 raft_consensus.cc:1261] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:01.256337 12585 raft_consensus.cc:1261] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:01.256700 21396 consensus_queue.cc:1035] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.256829 21292 consensus_queue.cc:1035] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.260121 12443 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet ffdc7e8f39c74b8887bd1e7431b19a10 (table rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1]). Expected version 1 got 0
I20240501 14:10:01.260481 21487 mvcc.cc:205] Tried to move back new op lower bound from 7022889374741196800 to 7022889374289944576. Current Snapshot: MvccSnapshot[applied={T|T < 7022889374741196800}]
I20240501 14:10:01.260725 12440 catalog_manager.cc:5576] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801) does not have the latest schema for tablet ffdc7e8f39c74b8887bd1e7431b19a10 (table rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1]). Expected version 1 got 0
I20240501 14:10:01.260624 21548 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:01.261837 21549 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:01.262194 21549 tablet.cc:1620] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:10:01.262471 12440 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet ffdc7e8f39c74b8887bd1e7431b19a10 (table rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1]). Expected version 1 got 0
I20240501 14:10:01.262923 21480 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:01.263165 21548 tablet.cc:1620] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:10:01.263217 21480 tablet.cc:1620] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:10:01.266933 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1] alter complete (version 1)
W20240501 14:10:01.270999 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:01.312650 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:01.359596 21364 consensus_queue.cc:1035] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.359874 21471 consensus_queue.cc:1035] T 9792f7aacce04968b694259014ceffbe P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.369223 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:10:01.374440 21548 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:01.374801 21548 tablet.cc:1620] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:10:01.377362 21480 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:01.377671 21480 tablet.cc:1620] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
W20240501 14:10:01.378239 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:01.380913 21549 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:01.382134 21549 tablet.cc:1620] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: MemRowSet was empty: no flush needed.
I20240501 14:10:01.382419 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=97a970871bd2452782890c37e2f710e1] alter complete (version 2)
I20240501 14:10:01.415441 21157 consensus_queue.cc:1035] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.419808 21473 consensus_queue.cc:1035] T d3dbe3b75f1d4c8d87e5acab12d2b515 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.484376 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:10:01.489440 21548 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:01.490000 21549 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:01.490525 21480 tablet.cc:1722] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:01.496693 12443 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=97a970871bd2452782890c37e2f710e1] alter complete (version 3)
I20240501 14:10:01.510293 20152 consensus_queue.cc:1035] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.513711 21056 consensus_queue.cc:1035] T 1f72db8c465f43fb9bef6e8c42303ea7 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.525141 12443 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:01.531635 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4fb99b88a94ecaa9e453bb124504c7 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.531630 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4fb99b88a94ecaa9e453bb124504c7 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.531945 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4fb99b88a94ecaa9e453bb124504c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.531955 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4fb99b88a94ecaa9e453bb124504c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.532485 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5f9dd560bb464ba80cbed95a477c2f (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.532737 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5f9dd560bb464ba80cbed95a477c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.534018 21499 tablet_bootstrap.cc:492] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:10:01.535027 21529 tablet_bootstrap.cc:492] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:10:01.535027 12696 tablet_service.cc:1467] Processing CreateTablet for tablet 44a2d42a2dc84baca7f54d45fd34d993 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.535300 12696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a2d42a2dc84baca7f54d45fd34d993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.535511 21571 tablet_bootstrap.cc:492] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:10:01.535588 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5f9dd560bb464ba80cbed95a477c2f (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.535786 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5f9dd560bb464ba80cbed95a477c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.535986 21529 tablet_bootstrap.cc:654] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.536422 21571 tablet_bootstrap.cc:654] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.536854 14938 tablet_service.cc:1467] Processing CreateTablet for tablet 0f4fb99b88a94ecaa9e453bb124504c7 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.537163 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f4fb99b88a94ecaa9e453bb124504c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.537575 21571 tablet_bootstrap.cc:492] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:10:01.537657 21571 ts_tablet_manager.cc:1367] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:01.538118 21571 raft_consensus.cc:348] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.538250 21571 raft_consensus.cc:374] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.538290 21571 raft_consensus.cc:729] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.538424 21571 consensus_queue.cc:260] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.539273 14937 tablet_service.cc:1467] Processing CreateTablet for tablet 44a2d42a2dc84baca7f54d45fd34d993 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.539502 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a2d42a2dc84baca7f54d45fd34d993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.539628 21499 tablet_bootstrap.cc:654] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.538714 21571 ts_tablet_manager.cc:1398] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.540071 21571 tablet_bootstrap.cc:492] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:10:01.540944 21571 tablet_bootstrap.cc:654] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.541051 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5f9dd560bb464ba80cbed95a477c2f (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.541338 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5f9dd560bb464ba80cbed95a477c2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.541707 15202 tablet_service.cc:1467] Processing CreateTablet for tablet 44a2d42a2dc84baca7f54d45fd34d993 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=88890a42e32b4793819c8870f1c7955c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:01.541944 15202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a2d42a2dc84baca7f54d45fd34d993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.542171 21576 tablet_bootstrap.cc:492] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:10:01.543568 21576 tablet_bootstrap.cc:654] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.544934 21576 tablet_bootstrap.cc:492] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:10:01.545651 21577 tablet_bootstrap.cc:492] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:01.545580 21576 ts_tablet_manager.cc:1367] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:01.546640 21577 tablet_bootstrap.cc:654] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.546599 21576 raft_consensus.cc:348] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.546871 21576 raft_consensus.cc:374] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.546953 21576 raft_consensus.cc:729] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.547128 21576 consensus_queue.cc:260] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.547179 21529 tablet_bootstrap.cc:492] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:10:01.547269 21529 ts_tablet_manager.cc:1367] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:10:01.547562 21576 ts_tablet_manager.cc:1398] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:01.547988 21571 tablet_bootstrap.cc:492] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:10:01.548063 21571 ts_tablet_manager.cc:1367] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 14:10:01.548520 21571 raft_consensus.cc:348] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.548622 21571 raft_consensus.cc:374] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.548657 21571 raft_consensus.cc:729] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.548784 21571 consensus_queue.cc:260] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.549077 21571 ts_tablet_manager.cc:1398] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.549254 21571 tablet_bootstrap.cc:492] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Bootstrap starting.
I20240501 14:10:01.549481 21577 tablet_bootstrap.cc:492] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:01.550129 21571 tablet_bootstrap.cc:654] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.550240 21529 raft_consensus.cc:348] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.550354 21529 raft_consensus.cc:374] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.550382 21529 raft_consensus.cc:729] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.550513 21529 consensus_queue.cc:260] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.551126 21529 ts_tablet_manager.cc:1398] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:10:01.552801 21499 tablet_bootstrap.cc:492] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:10:01.552902 21499 ts_tablet_manager.cc:1367] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:10:01.553545 21499 raft_consensus.cc:348] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.553651 21499 raft_consensus.cc:374] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.553683 21499 raft_consensus.cc:729] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.553793 21499 consensus_queue.cc:260] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.549552 21577 ts_tablet_manager.cc:1367] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 14:10:01.554080 21499 ts_tablet_manager.cc:1398] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.554442 21577 raft_consensus.cc:348] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.554548 21577 raft_consensus.cc:374] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.554584 21577 raft_consensus.cc:729] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.554703 21577 consensus_queue.cc:260] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.555042 21577 ts_tablet_manager.cc:1398] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.556159 21576 tablet_bootstrap.cc:492] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:10:01.556203 21577 tablet_bootstrap.cc:492] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:01.557228 21577 tablet_bootstrap.cc:654] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.558864 21498 raft_consensus.cc:482] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.558899 21577 tablet_bootstrap.cc:492] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:01.558984 21577 ts_tablet_manager.cc:1367] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:10:01.558959 21498 raft_consensus.cc:504] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.559031 21576 tablet_bootstrap.cc:654] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.559460 21498 leader_election.cc:288] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:10:01.559484 21577 raft_consensus.cc:348] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.559628 21577 raft_consensus.cc:374] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.559681 21577 raft_consensus.cc:729] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.559857 21577 consensus_queue.cc:260] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.559898 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fb99b88a94ecaa9e453bb124504c7" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:10:01.560134 21577 ts_tablet_manager.cc:1398] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.560252 21571 tablet_bootstrap.cc:492] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: No bootstrap required, opened a new log
I20240501 14:10:01.560353 21571 ts_tablet_manager.cc:1367] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20240501 14:10:01.560469 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fb99b88a94ecaa9e453bb124504c7" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:10:01.560595 12717 raft_consensus.cc:2454] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
W20240501 14:10:01.560863 15287 leader_election.cc:341] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:10:01.560855 21571 raft_consensus.cc:348] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.560954 21571 raft_consensus.cc:374] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.560988 21571 raft_consensus.cc:729] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ab42bc38694f7f9fc8bfcc0a1483b7, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.560966 15286 leader_election.cc:302] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters: 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:10:01.561113 21571 consensus_queue.cc:260] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.561211 21498 raft_consensus.cc:2781] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.561270 21498 raft_consensus.cc:482] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.561327 21498 raft_consensus.cc:3037] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.561405 21571 ts_tablet_manager.cc:1398] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:01.562270 21498 raft_consensus.cc:504] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.562640 21498 leader_election.cc:288] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177), e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113)
I20240501 14:10:01.562891 21576 tablet_bootstrap.cc:492] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:10:01.562968 21576 ts_tablet_manager.cc:1367] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:01.562938 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fb99b88a94ecaa9e453bb124504c7" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
W20240501 14:10:01.563330 15287 leader_election.cc:341] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:10:01.563997 21576 raft_consensus.cc:348] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.564441 21576 raft_consensus.cc:374] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.564525 21576 raft_consensus.cc:729] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.564831 21576 consensus_queue.cc:260] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.565112 21576 ts_tablet_manager.cc:1398] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:10:01.566469 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f4fb99b88a94ecaa9e453bb124504c7" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:10:01.566577 12717 raft_consensus.cc:3037] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.567693 12717 raft_consensus.cc:2454] T 0f4fb99b88a94ecaa9e453bb124504c7 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:10:01.568207 15286 leader_election.cc:302] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6ab42bc38694f7f9fc8bfcc0a1483b7, f23f426391724d6688aafffe7ee017d8; no voters: 8a13c028ba8a4a659b74af6c265df95e
I20240501 14:10:01.568401 21498 raft_consensus.cc:2781] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.568482 21498 raft_consensus.cc:686] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:10:01.568604 21498 consensus_queue.cc:237] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } }
I20240501 14:10:01.571106 12438 catalog_manager.cc:5484] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:01.599129 12438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:01.599289 12438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:01.601708 12438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet ffdc7e8f39c74b8887bd1e7431b19a10
I20240501 14:10:01.602380 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdc7e8f39c74b8887bd1e7431b19a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:10:01.602541 15331 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdc7e8f39c74b8887bd1e7431b19a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:44450
I20240501 14:10:01.602898 12959 tablet_service.cc:1514] Processing DeleteTablet for tablet ffdc7e8f39c74b8887bd1e7431b19a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:01 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:10:01.602939 21585 tablet_replica.cc:330] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: stopping tablet replica
I20240501 14:10:01.603135 21585 raft_consensus.cc:2229] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.603327 21585 raft_consensus.cc:2258] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.603643 21586 tablet_replica.cc:330] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:10:01.603811 21586 raft_consensus.cc:2229] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:01.604068 21586 raft_consensus.cc:2258] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.604903 21586 ts_tablet_manager.cc:1875] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.604957 21585 ts_tablet_manager.cc:1875] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.607354 21586 ts_tablet_manager.cc:1888] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.607445 21586 log.cc:1196] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/ffdc7e8f39c74b8887bd1e7431b19a10
I20240501 14:10:01.607509 21585 ts_tablet_manager.cc:1888] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.607616 21585 log.cc:1196] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-7/wal/wals/ffdc7e8f39c74b8887bd1e7431b19a10
I20240501 14:10:01.607751 21586 ts_tablet_manager.cc:1909] T ffdc7e8f39c74b8887bd1e7431b19a10 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:10:01.607964 21585 ts_tablet_manager.cc:1909] T ffdc7e8f39c74b8887bd1e7431b19a10 P f23f426391724d6688aafffe7ee017d8: Deleting consensus metadata
I20240501 14:10:01.608592 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet ffdc7e8f39c74b8887bd1e7431b19a10 (table rebalancer_dynamic_table_new_name [id=97a970871bd2452782890c37e2f710e1]) successfully deleted
I20240501 14:10:01.608919 12432 catalog_manager.cc:4830] TS f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): tablet ffdc7e8f39c74b8887bd1e7431b19a10 (table rebalancer_dynamic_table_new_name [id=97a970871bd2452782890c37e2f710e1]) successfully deleted
I20240501 14:10:01.614125 21584 tablet_replica.cc:330] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:10:01.614302 21584 raft_consensus.cc:2229] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:01.614434 21584 raft_consensus.cc:2258] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:01.615227 21584 ts_tablet_manager.cc:1875] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:01.617830 21584 ts_tablet_manager.cc:1888] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:01.617923 21584 log.cc:1196] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/ffdc7e8f39c74b8887bd1e7431b19a10
I20240501 14:10:01.618357 21584 ts_tablet_manager.cc:1909] T ffdc7e8f39c74b8887bd1e7431b19a10 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:10:01.619118 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet ffdc7e8f39c74b8887bd1e7431b19a10 (table rebalancer_dynamic_table_new_name [id=97a970871bd2452782890c37e2f710e1]) successfully deleted
I20240501 14:10:01.679654 21038 raft_consensus.cc:482] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.679761 21038 raft_consensus.cc:504] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.680122 21038 leader_election.cc:288] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:01.680446 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5f9dd560bb464ba80cbed95a477c2f" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:10:01.680546 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5f9dd560bb464ba80cbed95a477c2f" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:10:01.680591 12717 raft_consensus.cc:2454] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:10:01.680680 15220 raft_consensus.cc:2454] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:10:01.680886 12520 leader_election.cc:302] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:01.681157 21038 raft_consensus.cc:2781] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.681218 21038 raft_consensus.cc:482] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.681259 21038 raft_consensus.cc:3037] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.682341 21038 raft_consensus.cc:504] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.682628 21038 leader_election.cc:288] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:01.682957 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5f9dd560bb464ba80cbed95a477c2f" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:10:01.683019 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5f9dd560bb464ba80cbed95a477c2f" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:10:01.683082 12717 raft_consensus.cc:3037] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.683154 15220 raft_consensus.cc:3037] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.684100 15220 raft_consensus.cc:2454] T 2a5f9dd560bb464ba80cbed95a477c2f P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:10:01.684100 12717 raft_consensus.cc:2454] T 2a5f9dd560bb464ba80cbed95a477c2f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:10:01.684383 12520 leader_election.cc:302] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:10:01.684581 21038 raft_consensus.cc:2781] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.684749 21038 raft_consensus.cc:686] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:10:01.684899 21038 consensus_queue.cc:237] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.687886 12438 catalog_manager.cc:5484] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:01.704671 12438 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 14:10:01.710541 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd60af7c2a044c397b074d4553786c2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]), partition=
I20240501 14:10:01.710781 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd60af7c2a044c397b074d4553786c2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]), partition=
I20240501 14:10:01.710872 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd60af7c2a044c397b074d4553786c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.710845 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 1bd60af7c2a044c397b074d4553786c2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]), partition=
I20240501 14:10:01.711033 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd60af7c2a044c397b074d4553786c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.711066 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bd60af7c2a044c397b074d4553786c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:01.712844 21529 tablet_bootstrap.cc:492] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:10:01.713299 21577 tablet_bootstrap.cc:492] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:01.714094 21577 tablet_bootstrap.cc:654] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.714156 21529 tablet_bootstrap.cc:654] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.715093 21577 tablet_bootstrap.cc:492] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:01.715181 21577 ts_tablet_manager.cc:1367] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:10:01.715636 21577 raft_consensus.cc:348] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.715759 21577 raft_consensus.cc:374] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.715792 21577 raft_consensus.cc:729] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.715919 21577 consensus_queue.cc:260] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.716210 21577 ts_tablet_manager.cc:1398] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.716286 21529 tablet_bootstrap.cc:492] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:10:01.716365 21529 ts_tablet_manager.cc:1367] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:10:01.716810 21529 raft_consensus.cc:348] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.716929 21529 raft_consensus.cc:374] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.716972 21529 raft_consensus.cc:729] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.717092 21529 consensus_queue.cc:260] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.717365 21529 ts_tablet_manager.cc:1398] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:01.717497 21600 tablet_bootstrap.cc:492] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:01.718441 21600 tablet_bootstrap.cc:654] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:01.719952 21600 tablet_bootstrap.cc:492] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:01.720108 21600 ts_tablet_manager.cc:1367] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:10:01.721189 21600 raft_consensus.cc:348] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.721379 21600 raft_consensus.cc:374] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:01.721442 21600 raft_consensus.cc:729] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:01.721767 21600 consensus_queue.cc:260] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.722157 21600 ts_tablet_manager.cc:1398] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:01.729854 21364 raft_consensus.cc:482] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.729982 21364 raft_consensus.cc:504] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.730448 21364 leader_election.cc:288] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:01.730896 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a2d42a2dc84baca7f54d45fd34d993" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" is_pre_election: true
I20240501 14:10:01.730896 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a2d42a2dc84baca7f54d45fd34d993" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:10:01.731038 15220 raft_consensus.cc:2454] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:10:01.731038 12717 raft_consensus.cc:2454] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 0.
I20240501 14:10:01.731359 14891 leader_election.cc:302] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 8a13c028ba8a4a659b74af6c265df95e; no voters:
I20240501 14:10:01.731683 21364 raft_consensus.cc:2781] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.731770 21364 raft_consensus.cc:482] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.731820 21364 raft_consensus.cc:3037] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.732748 21364 raft_consensus.cc:504] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.733016 21364 leader_election.cc:288] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Requested vote from peers e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:01.733352 12717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a2d42a2dc84baca7f54d45fd34d993" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7"
I20240501 14:10:01.733469 12717 raft_consensus.cc:3037] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.733695 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a2d42a2dc84baca7f54d45fd34d993" candidate_uuid: "8a13c028ba8a4a659b74af6c265df95e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:10:01.733814 15220 raft_consensus.cc:3037] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.734544 12717 raft_consensus.cc:2454] T 44a2d42a2dc84baca7f54d45fd34d993 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:10:01.734794 15220 raft_consensus.cc:2454] T 44a2d42a2dc84baca7f54d45fd34d993 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a13c028ba8a4a659b74af6c265df95e in term 1.
I20240501 14:10:01.734830 14891 leader_election.cc:302] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, e6ab42bc38694f7f9fc8bfcc0a1483b7; no voters:
I20240501 14:10:01.735266 21364 raft_consensus.cc:2781] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.735354 21364 raft_consensus.cc:686] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Becoming Leader. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Running, Role: LEADER
I20240501 14:10:01.735484 21364 consensus_queue.cc:237] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.738220 12443 catalog_manager.cc:5484] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5). New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:01.803668 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:10:01.882562 21292 raft_consensus.cc:482] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:01.882678 21292 raft_consensus.cc:504] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.883095 21292 leader_election.cc:288] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:01.883388 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd60af7c2a044c397b074d4553786c2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" is_pre_election: true
I20240501 14:10:01.883528 12585 raft_consensus.cc:2454] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:10:01.883558 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd60af7c2a044c397b074d4553786c2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:10:01.883829 12915 leader_election.cc:302] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:01.883978 21292 raft_consensus.cc:2781] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:01.884035 21292 raft_consensus.cc:482] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:01.884083 21292 raft_consensus.cc:3037] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.884057 15220 raft_consensus.cc:2454] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 0.
I20240501 14:10:01.885179 21292 raft_consensus.cc:504] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.885505 21292 leader_election.cc:288] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Requested vote from peers 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:01.885787 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd60af7c2a044c397b074d4553786c2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a3cc0660eed427eaee38164a8c1d4c3"
I20240501 14:10:01.885918 12585 raft_consensus.cc:3037] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.886925 12585 raft_consensus.cc:2454] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:10:01.886283 15220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bd60af7c2a044c397b074d4553786c2" candidate_uuid: "a6e94562bee149c2ace25672f0f9bf50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:10:01.887113 15220 raft_consensus.cc:3037] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.887185 12915 leader_election.cc:302] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:01.887363 21292 raft_consensus.cc:2781] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:01.888299 15220 raft_consensus.cc:2454] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e94562bee149c2ace25672f0f9bf50 in term 1.
I20240501 14:10:01.889261 21292 raft_consensus.cc:686] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Becoming Leader. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Running, Role: LEADER
I20240501 14:10:01.889475 21292 consensus_queue.cc:237] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:01.892438 12443 catalog_manager.cc:5484] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4). New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:01.902519 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:01.959586 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:01.964203 14957 raft_consensus.cc:3037] T 0f4fb99b88a94ecaa9e453bb124504c7 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:01.965524 21366 consensus_queue.cc:1035] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:01.968797 21366 consensus_queue.cc:1035] T 0f4fb99b88a94ecaa9e453bb124504c7 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:10:01.975607 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.002288 12443 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 14:10:02.006379 15220 raft_consensus.cc:1261] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:02.006661 12585 raft_consensus.cc:1261] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:10:02.007133 21292 consensus_queue.cc:1035] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.007468 21396 consensus_queue.cc:1035] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.010838 12443 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]). Expected version 1 got 0
I20240501 14:10:02.011766 12438 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]). Expected version 1 got 0
I20240501 14:10:02.012163 21626 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:02.013692 21626 tablet.cc:1620] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:10:02.014047 21627 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:02.014569 21627 tablet.cc:1620] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:10:02.014840 12438 catalog_manager.cc:5576] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151) does not have the latest schema for tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]). Expected version 1 got 0
I20240501 14:10:02.016021 12440 catalog_manager.cc:5576] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847) does not have the latest schema for tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]). Expected version 1 got 0
I20240501 14:10:02.019948 12443 catalog_manager.cc:5576] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767) does not have the latest schema for tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441]). Expected version 1 got 0
I20240501 14:10:02.020637 21628 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 14:10:02.020417 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441] alter complete (version 1)
I20240501 14:10:02.021289 21628 tablet.cc:1620] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
W20240501 14:10:02.027236 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.049957 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:02.054642 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.123441 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 14:10:02.128710 21626 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:02.129046 21626 tablet.cc:1620] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: MemRowSet was empty: no flush needed.
I20240501 14:10:02.130178 21628 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:02.130200 21627 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 14:10:02.130484 21627 tablet.cc:1620] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: MemRowSet was empty: no flush needed.
I20240501 14:10:02.130568 21628 tablet.cc:1620] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: MemRowSet was empty: no flush needed.
I20240501 14:10:02.134624 12440 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table [id=f7bd6043d06e4492b9f1cd2ea7246441] alter complete (version 2)
W20240501 14:10:02.160015 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.166659 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:10:02.167236 21364 consensus_queue.cc:1035] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.170643 21364 consensus_queue.cc:1035] T 44a2d42a2dc84baca7f54d45fd34d993 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.216622 21038 consensus_queue.cc:1035] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.217612 21038 consensus_queue.cc:1035] T 2a5f9dd560bb464ba80cbed95a477c2f P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.236214 12440 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 14:10:02.241457 21626 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:02.241945 21627 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:02.244880 21628 tablet.cc:1722] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 14:10:02.247464 12442 catalog_manager.cc:6029] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: rebalancer_dynamic_table_new_name [id=f7bd6043d06e4492b9f1cd2ea7246441] alter complete (version 3)
I20240501 14:10:02.281831 12440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "rebalancer_extra_table_76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:10:02.289160 12959 tablet_service.cc:1467] Processing CreateTablet for tablet 96487ed1deac4783a9b50c6b05f843c5 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.289304 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 96487ed1deac4783a9b50c6b05f843c5 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.289537 12959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96487ed1deac4783a9b50c6b05f843c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.289613 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96487ed1deac4783a9b50c6b05f843c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.290292 14938 tablet_service.cc:1467] Processing CreateTablet for tablet bce72645bce042319763dadb66624a72 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.290644 14938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce72645bce042319763dadb66624a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.291285 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 430160d775a744b49780d4ab83db4123 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.291463 15201 tablet_service.cc:1467] Processing CreateTablet for tablet 430160d775a744b49780d4ab83db4123 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.291549 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430160d775a744b49780d4ab83db4123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.291743 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430160d775a744b49780d4ab83db4123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.292382 12958 tablet_service.cc:1467] Processing CreateTablet for tablet 430160d775a744b49780d4ab83db4123 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.292610 12958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430160d775a744b49780d4ab83db4123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.294186 12829 tablet_service.cc:1467] Processing CreateTablet for tablet 96487ed1deac4783a9b50c6b05f843c5 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.294471 12829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96487ed1deac4783a9b50c6b05f843c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.294942 12827 tablet_service.cc:1467] Processing CreateTablet for tablet bce72645bce042319763dadb66624a72 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.295151 12827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce72645bce042319763dadb66624a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.295210 21657 tablet_bootstrap.cc:492] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8: Bootstrap starting.
I20240501 14:10:02.295276 21658 tablet_bootstrap.cc:492] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:02.296252 21658 tablet_bootstrap.cc:654] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.296955 21656 tablet_bootstrap.cc:492] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3: Bootstrap starting.
I20240501 14:10:02.298197 21657 tablet_bootstrap.cc:654] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.298363 21656 tablet_bootstrap.cc:654] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.299351 15331 tablet_service.cc:1467] Processing CreateTablet for tablet bce72645bce042319763dadb66624a72 (DEFAULT_TABLE table=rebalancer_extra_table_76 [id=1bc71fe7c218407588b07376afaf7e16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:10:02.299674 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce72645bce042319763dadb66624a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:02.300122 21657 tablet_bootstrap.cc:492] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8: No bootstrap required, opened a new log
I20240501 14:10:02.300251 21657 ts_tablet_manager.cc:1367] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:10:02.300768 21664 tablet_bootstrap.cc:492] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e: Bootstrap starting.
I20240501 14:10:02.300778 21657 raft_consensus.cc:348] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.300973 21657 raft_consensus.cc:374] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.301061 21657 raft_consensus.cc:729] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e2dc34330848c3b8c2dd15898d34c8, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.301362 21657 consensus_queue.cc:260] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.301739 21657 ts_tablet_manager.cc:1398] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:10:02.301853 21664 tablet_bootstrap.cc:654] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.302835 21656 tablet_bootstrap.cc:492] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3: No bootstrap required, opened a new log
I20240501 14:10:02.302923 21656 ts_tablet_manager.cc:1367] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240501 14:10:02.303325 21656 raft_consensus.cc:348] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.303455 21656 raft_consensus.cc:374] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.303483 21656 raft_consensus.cc:729] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.303591 21656 consensus_queue.cc:260] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.303889 21656 ts_tablet_manager.cc:1398] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 14:10:02.303915 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:10:02.304008 21655 tablet_bootstrap.cc:492] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:10:02.304937 21655 tablet_bootstrap.cc:654] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.306118 21664 tablet_bootstrap.cc:492] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e: No bootstrap required, opened a new log
I20240501 14:10:02.306206 21664 ts_tablet_manager.cc:1367] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:10:02.306653 21664 raft_consensus.cc:348] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.306779 21664 raft_consensus.cc:374] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.306818 21664 raft_consensus.cc:729] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a13c028ba8a4a659b74af6c265df95e, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.306933 21664 consensus_queue.cc:260] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.307229 21664 ts_tablet_manager.cc:1398] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:02.307806 21660 tablet_bootstrap.cc:492] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:02.308756 21660 tablet_bootstrap.cc:654] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.309863 21658 tablet_bootstrap.cc:492] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:02.309955 21658 ts_tablet_manager.cc:1367] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 14:10:02.310037 21660 tablet_bootstrap.cc:492] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:02.310112 21660 ts_tablet_manager.cc:1367] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:02.310422 21658 raft_consensus.cc:348] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.310451 21660 raft_consensus.cc:348] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.310528 21658 raft_consensus.cc:374] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.310536 21660 raft_consensus.cc:374] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.310564 21658 raft_consensus.cc:729] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.310568 21660 raft_consensus.cc:729] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.310685 21660 consensus_queue.cc:260] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.310685 21658 consensus_queue.cc:260] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.310995 21660 ts_tablet_manager.cc:1398] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:02.310995 21658 ts_tablet_manager.cc:1398] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:02.311157 21660 tablet_bootstrap.cc:492] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:10:02.311175 21658 tablet_bootstrap.cc:492] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50: Bootstrap starting.
I20240501 14:10:02.311928 21660 tablet_bootstrap.cc:654] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.311998 21658 tablet_bootstrap.cc:654] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.312798 21655 tablet_bootstrap.cc:492] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:10:02.312911 21655 ts_tablet_manager.cc:1367] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:10:02.313336 21655 raft_consensus.cc:348] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.313445 21655 raft_consensus.cc:374] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.313517 21655 raft_consensus.cc:729] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.313654 21655 consensus_queue.cc:260] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.314900 21655 ts_tablet_manager.cc:1398] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:02.315066 21655 tablet_bootstrap.cc:492] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8: Bootstrap starting.
I20240501 14:10:02.316879 21655 tablet_bootstrap.cc:654] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8: Neither blocks nor log segments found. Creating new log.
I20240501 14:10:02.317838 21658 tablet_bootstrap.cc:492] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50: No bootstrap required, opened a new log
I20240501 14:10:02.317977 21658 ts_tablet_manager.cc:1367] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:10:02.318990 21658 raft_consensus.cc:348] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.319169 21658 raft_consensus.cc:374] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.319219 21658 raft_consensus.cc:729] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e94562bee149c2ace25672f0f9bf50, State: Initialized, Role: FOLLOWER
W20240501 14:10:02.319351 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.319365 21658 consensus_queue.cc:260] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.319734 21658 ts_tablet_manager.cc:1398] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:02.320389 21660 tablet_bootstrap.cc:492] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2: No bootstrap required, opened a new log
I20240501 14:10:02.320523 21660 ts_tablet_manager.cc:1367] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:10:02.320536 21655 tablet_bootstrap.cc:492] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8: No bootstrap required, opened a new log
I20240501 14:10:02.320606 21655 ts_tablet_manager.cc:1367] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 14:10:02.321096 21655 raft_consensus.cc:348] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.321146 21660 raft_consensus.cc:348] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.321257 21660 raft_consensus.cc:374] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.321277 21655 raft_consensus.cc:374] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:10:02.321327 21660 raft_consensus.cc:729] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.321363 21655 raft_consensus.cc:729] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Initialized, Role: FOLLOWER
I20240501 14:10:02.321535 21655 consensus_queue.cc:260] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.321563 21660 consensus_queue.cc:260] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.321867 21655 ts_tablet_manager.cc:1398] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:10:02.322964 21660 ts_tablet_manager.cc:1398] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 14:10:02.328655 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.348991 12443 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:02.349170 12443 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 14:10:02.351300 12443 catalog_manager.cc:5771] T 00000000000000000000000000000000 P bd843b84065446c09f71ac8534f21a8a: Sending DeleteTablet for 3 replicas of tablet 1bd60af7c2a044c397b074d4553786c2
I20240501 14:10:02.351835 12958 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd60af7c2a044c397b074d4553786c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:02 UTC) from {username='slave'} at 127.0.0.1:57202
I20240501 14:10:02.351882 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd60af7c2a044c397b074d4553786c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:02 UTC) from {username='slave'} at 127.0.0.1:35710
I20240501 14:10:02.352156 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bd60af7c2a044c397b074d4553786c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 14:10:02 UTC) from {username='slave'} at 127.0.0.1:60582
I20240501 14:10:02.352209 21671 tablet_replica.cc:330] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: stopping tablet replica
I20240501 14:10:02.352363 21671 raft_consensus.cc:2229] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:10:02.352576 21671 raft_consensus.cc:2258] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:02.352818 21672 tablet_replica.cc:330] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: stopping tablet replica
I20240501 14:10:02.353327 21671 ts_tablet_manager.cc:1875] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:02.353446 21673 tablet_replica.cc:330] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: stopping tablet replica
I20240501 14:10:02.353662 21673 raft_consensus.cc:2229] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:02.353849 21673 raft_consensus.cc:2258] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:02.354368 21673 ts_tablet_manager.cc:1875] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 14:10:02.356299 21673 ts_tablet_manager.cc:1888] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:02.356434 21673 log.cc:1196] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-6/wal/wals/1bd60af7c2a044c397b074d4553786c2
I20240501 14:10:02.356730 21673 ts_tablet_manager.cc:1909] T 1bd60af7c2a044c397b074d4553786c2 P 66e2dc34330848c3b8c2dd15898d34c8: Deleting consensus metadata
I20240501 14:10:02.356971 21671 ts_tablet_manager.cc:1888] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:02.357049 21671 log.cc:1196] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-3/wal/wals/1bd60af7c2a044c397b074d4553786c2
I20240501 14:10:02.357352 21671 ts_tablet_manager.cc:1909] T 1bd60af7c2a044c397b074d4553786c2 P a6e94562bee149c2ace25672f0f9bf50: Deleting consensus metadata
I20240501 14:10:02.357573 12429 catalog_manager.cc:4830] TS 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847): tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table_new_name [id=f7bd6043d06e4492b9f1cd2ea7246441]) successfully deleted
I20240501 14:10:02.357976 12431 catalog_manager.cc:4830] TS a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table_new_name [id=f7bd6043d06e4492b9f1cd2ea7246441]) successfully deleted
I20240501 14:10:02.358006 21672 raft_consensus.cc:2229] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:10:02.358161 21672 raft_consensus.cc:2258] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:10:02.359077 21672 ts_tablet_manager.cc:1875] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 14:10:02.359390 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.361171 21672 ts_tablet_manager.cc:1888] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 14:10:02.361243 21672 log.cc:1196] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting WAL directory at /tmp/dist-test-taskrG4Tzf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714572530403361-6472-0/raft_consensus-itest-cluster/ts-0/wal/wals/1bd60af7c2a044c397b074d4553786c2
I20240501 14:10:02.361537 21672 ts_tablet_manager.cc:1909] T 1bd60af7c2a044c397b074d4553786c2 P 6a3cc0660eed427eaee38164a8c1d4c3: Deleting consensus metadata
I20240501 14:10:02.362149 12431 catalog_manager.cc:4830] TS 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): tablet 1bd60af7c2a044c397b074d4553786c2 (table rebalancer_dynamic_table_new_name [id=f7bd6043d06e4492b9f1cd2ea7246441]) successfully deleted
I20240501 14:10:02.416574 21498 raft_consensus.cc:482] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:02.416707 21498 raft_consensus.cc:504] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.417160 21498 leader_election.cc:288] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
I20240501 14:10:02.417569 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce72645bce042319763dadb66624a72" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e" is_pre_election: true
I20240501 14:10:02.417622 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce72645bce042319763dadb66624a72" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2" is_pre_election: true
I20240501 14:10:02.417742 14957 raft_consensus.cc:2454] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:10:02.417791 12849 raft_consensus.cc:2454] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 0.
I20240501 14:10:02.418064 15287 leader_election.cc:302] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a13c028ba8a4a659b74af6c265df95e, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:10:02.418292 21498 raft_consensus.cc:2781] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:02.418359 21498 raft_consensus.cc:482] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:02.418401 21498 raft_consensus.cc:3037] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.419618 21498 raft_consensus.cc:504] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.420428 12849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce72645bce042319763dadb66624a72" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4902d2609e6d44d089590a42e70356b2"
I20240501 14:10:02.420562 12849 raft_consensus.cc:3037] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.420828 21498 leader_election.cc:288] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Requested vote from peers 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215), 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177)
W20240501 14:10:02.421214 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240501 14:10:02.421664 12849 raft_consensus.cc:2454] T bce72645bce042319763dadb66624a72 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:10:02.421746 14957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce72645bce042319763dadb66624a72" candidate_uuid: "f23f426391724d6688aafffe7ee017d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a13c028ba8a4a659b74af6c265df95e"
I20240501 14:10:02.421856 14957 raft_consensus.cc:3037] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.422128 15289 leader_election.cc:302] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, f23f426391724d6688aafffe7ee017d8; no voters:
I20240501 14:10:02.422892 14957 raft_consensus.cc:2454] T bce72645bce042319763dadb66624a72 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23f426391724d6688aafffe7ee017d8 in term 1.
I20240501 14:10:02.423009 21617 raft_consensus.cc:2781] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:02.423092 21617 raft_consensus.cc:686] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Becoming Leader. State: Replica: f23f426391724d6688aafffe7ee017d8, State: Running, Role: LEADER
I20240501 14:10:02.423226 21617 consensus_queue.cc:237] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } }
I20240501 14:10:02.425709 12443 catalog_manager.cc:5484] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23f426391724d6688aafffe7ee017d8 (127.6.82.8). New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:02.461612 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.473884 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.473937 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:02.495432 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.576270 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:02.600083 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:10:02.604601 21641 raft_consensus.cc:482] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:02.604722 21641 raft_consensus.cc:504] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.605170 21641 leader_election.cc:288] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:02.605527 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430160d775a744b49780d4ab83db4123" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:10:02.605667 12978 raft_consensus.cc:2454] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:10:02.605654 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430160d775a744b49780d4ab83db4123" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8" is_pre_election: true
I20240501 14:10:02.605821 15221 raft_consensus.cc:2454] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 0.
I20240501 14:10:02.606132 12520 leader_election.cc:302] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66e2dc34330848c3b8c2dd15898d34c8, 6a3cc0660eed427eaee38164a8c1d4c3; no voters:
I20240501 14:10:02.607077 21641 raft_consensus.cc:2781] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:02.607152 21641 raft_consensus.cc:482] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:02.607196 21641 raft_consensus.cc:3037] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.608202 21641 raft_consensus.cc:504] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.608852 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430160d775a744b49780d4ab83db4123" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:10:02.609007 12978 raft_consensus.cc:3037] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.609297 21641 leader_election.cc:288] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), 66e2dc34330848c3b8c2dd15898d34c8 (127.6.82.7:34847)
I20240501 14:10:02.610008 12978 raft_consensus.cc:2454] T 430160d775a744b49780d4ab83db4123 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:10:02.610221 15221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430160d775a744b49780d4ab83db4123" candidate_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66e2dc34330848c3b8c2dd15898d34c8"
I20240501 14:10:02.610330 15221 raft_consensus.cc:3037] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.611032 12522 leader_election.cc:302] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a3cc0660eed427eaee38164a8c1d4c3, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:02.611210 15221 raft_consensus.cc:2454] T 430160d775a744b49780d4ab83db4123 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a3cc0660eed427eaee38164a8c1d4c3 in term 1.
I20240501 14:10:02.611279 21641 raft_consensus.cc:2781] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:02.611377 21641 raft_consensus.cc:686] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3cc0660eed427eaee38164a8c1d4c3, State: Running, Role: LEADER
I20240501 14:10:02.611511 21641 consensus_queue.cc:237] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } }
I20240501 14:10:02.614089 12443 catalog_manager.cc:5484] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1). New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:02.619979 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.676870 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.686252 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:10:02.686970 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.720963 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.727643 21550 raft_consensus.cc:482] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:02.727790 21550 raft_consensus.cc:504] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.728241 21550 leader_election.cc:288] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:10:02.728546 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96487ed1deac4783a9b50c6b05f843c5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50" is_pre_election: true
I20240501 14:10:02.728617 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96487ed1deac4783a9b50c6b05f843c5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8" is_pre_election: true
I20240501 14:10:02.728713 12978 raft_consensus.cc:2454] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:10:02.728749 15351 raft_consensus.cc:2454] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 0.
I20240501 14:10:02.729029 12784 leader_election.cc:302] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:02.729202 21550 raft_consensus.cc:2781] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:10:02.729259 21550 raft_consensus.cc:482] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:10:02.729301 21550 raft_consensus.cc:3037] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.730271 21550 raft_consensus.cc:504] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.730566 21550 leader_election.cc:288] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Requested vote from peers a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151), f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801)
I20240501 14:10:02.730883 12978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96487ed1deac4783a9b50c6b05f843c5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e94562bee149c2ace25672f0f9bf50"
I20240501 14:10:02.730938 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96487ed1deac4783a9b50c6b05f843c5" candidate_uuid: "4902d2609e6d44d089590a42e70356b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23f426391724d6688aafffe7ee017d8"
I20240501 14:10:02.731040 15351 raft_consensus.cc:3037] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.731042 12978 raft_consensus.cc:3037] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:10:02.732084 15351 raft_consensus.cc:2454] T 96487ed1deac4783a9b50c6b05f843c5 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:10:02.732185 12978 raft_consensus.cc:2454] T 96487ed1deac4783a9b50c6b05f843c5 P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4902d2609e6d44d089590a42e70356b2 in term 1.
I20240501 14:10:02.732693 12784 leader_election.cc:302] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4902d2609e6d44d089590a42e70356b2, a6e94562bee149c2ace25672f0f9bf50; no voters:
I20240501 14:10:02.732862 21550 raft_consensus.cc:2781] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:10:02.733023 21550 raft_consensus.cc:686] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Running, Role: LEADER
I20240501 14:10:02.733162 21550 consensus_queue.cc:237] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } }
I20240501 14:10:02.736415 12443 catalog_manager.cc:5484] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4902d2609e6d44d089590a42e70356b2 (127.6.82.3). New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } }
W20240501 14:10:02.743870 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:02.770766 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:02.823618 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:02.829877 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:10:02.832782 21366 consensus_queue.cc:1035] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:02.837997 21366 consensus_queue.cc:1035] T bce72645bce042319763dadb66624a72 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:03.037616 21038 consensus_queue.cc:1035] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:03.042071 21717 consensus_queue.cc:1035] T 430160d775a744b49780d4ab83db4123 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:10:03.065831 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:03.111562 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:10:03.118631 21550 consensus_queue.cc:1035] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:10:03.122081 21723 consensus_queue.cc:1035] T 96487ed1deac4783a9b50c6b05f843c5 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:10:03.336519 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:03.551015 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:03.562021 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:03.630452 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:03.671491 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:03.769737 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:03.912425 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:03.925014 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.025975 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.395814 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.397378 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.484923 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:04.515010 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.570362 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.642957 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.661047 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.674974 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:04.675845 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.706974 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.723261 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:10:04.965453 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:04.982779 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:04.986773 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.000846 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.121066 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.239120 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.280624 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.284016 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.333739 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.456264 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.471343 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.493505 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:05.828588 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.885107 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:05.973183 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:06.127498 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:06.135758 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:06.386497 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:06.414342 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:06.599040 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:06.625480 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:06.848680 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:06.955791 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.013780 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.015300 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:07.096540 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.109074 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:10:07.138435 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.152581 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.186755 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.219470 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.252769 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.335036 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.461848 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:07.500769 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:07.559197 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.580338 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.603448 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:07.611878 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.635668 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:07.640542 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.686802 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:10:07.687641 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.763895 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:07.841884 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:07.916666 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:07.928241 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.012641 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.072801 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:08.192302 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:08.192795 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.319264 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.555543 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.678072 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.691427 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:08.861780 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:08.868460 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:08.879200 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:09.188031 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:09.273279 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.315848 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.357055 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.520641 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.523517 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:09.559137 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.573482 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.635263 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:09.667200 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.669663 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:10:09.743057 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.812927 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.955374 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.956638 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:09.977128 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.063767 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.092398 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:10.099649 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.264089 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.278949 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.456720 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.526422 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.623692 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:10.652633 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:10.707967 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:11.065006 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:11.092809 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:11.200126 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:11.296080 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:11.451153 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:11.515719 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:11.604223 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:11.763211 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:11.825523 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:11.925567 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:11.956446 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:12.014066 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.083065 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.115975 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.117626 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:10:12.131918 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.179109 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:12.233428 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.446467 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.460500 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.508014 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:12.578244 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.584662 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.661013 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:12.662717 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.734465 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:12.741832 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:10:12.742442 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.756458 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.799837 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:12.800590 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.813948 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:12.827175 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:13.006948 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:13.057826 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:13.060904 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:13.167351 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:13.213820 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 14:10:13.344333 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:13.363804 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:13.570034 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:13.580376 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:13.730016 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:13.912480 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 14:10:13.913412 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): CreateTablet RPC failed for tablet c0bf116648874fc3a3ff54c7addf5e2d: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:13.913496 12431 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet c0bf116648874fc3a3ff54c7addf5e2d on TS ab90e77f0027449bb3155980ca95593c
W20240501 14:10:13.913601 12431 catalog_manager.cc:4605] Request timed out: CreateTablet RPC for tablet 6f6e2acdbdc747a8a863d6e46d0d1b41 on TS ab90e77f0027449bb3155980ca95593c
W20240501 14:10:14.046186 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:14.061883 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:14.170708 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.214947 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.296840 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:14.422574 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.432958 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:14.473399 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.569734 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.570694 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:10:14.574323 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.593904 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.638507 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.695495 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:14.939333 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.012430 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.016743 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.033177 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:15.060773 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.144579 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.179773 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.225374 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:15.231945 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.326118 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.332693 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.433683 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.575953 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:15.579639 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.683866 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:15.959410 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:16.040825 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:16.134970 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:16.314647 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:16.376868 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:16.491777 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:16.655064 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:16.670656 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:16.730718 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:16.800027 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:16.902907 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:16.974568 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:16.977579 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:16.999364 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:17.074795 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.088999 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:10:17.091974 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.173066 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.293677 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.380303 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.425734 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:17.557582 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:17.561755 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.643941 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.686810 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.709702 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.732466 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:17.760473 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.806463 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:17.822724 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:10:17.823663 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.839943 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:17.895277 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:18.023236 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.026928 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:18.039846 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.097220 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.125993 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.222823 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:18.358850 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:18.443967 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.477421 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:10:18.722889 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.815675 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:18.883592 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:19.061619 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.084124 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:19.128863 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.195349 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.335494 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:19.336789 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.406056 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:19.428154 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.462890 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.613422 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.659121 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.731077 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.744705 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:10:19.777340 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.818198 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:19.838985 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:19.935652 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.136649 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.215662 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:20.243928 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.277266 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.327664 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.397171 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.452530 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.493108 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.515246 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.525983 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:20.660650 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:21.102980 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:21.141052 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:21.352361 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:21.391224 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:21.405174 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:21.461396 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:21.465570 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:21.716490 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:21.748303 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:21.749468 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:21.947540 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:21.948413 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.012413 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.134912 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.184309 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:22.193907 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:10:22.287706 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.313058 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.331876 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.429395 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.576539 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.579505 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:22.598100 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.644843 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.652701 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:22.761409 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.815912 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.844967 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:10:22.874531 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:10:22.896873 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:10:22.897497 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.899883 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:22.956547 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:23.048574 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:23.085703 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:23.141283 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:23.241278 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:23.449064 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:10:23.504124 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:23.536070 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:23.635272 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:23.677031 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:23.717103 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:23.747236 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:23.803591 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:23.898154 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:24.036929 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:24.199744 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:24.276630 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.323226 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.413250 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:24.502765 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.551424 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.576340 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.586629 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.700466 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.731876 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:10:24.805269 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:24.880980 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.910626 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:24.953480 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.146835 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.255846 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.344542 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.349467 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.377650 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.389837 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:25.427203 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.560271 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.594444 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:25.815034 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:26.005978 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:26.163812 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:26.164103 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:26.260887 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:26.265787 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:26.380750 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:26.421738 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:26.525209 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:26.686666 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:26.883975 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:26.887120 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:26.952023 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:26.961526 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.050966 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.095098 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:10:27.136860 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.157414 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.186483 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.339303 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.409554 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:27.430478 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.548403 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.570083 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.578047 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.600452 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:27.681517 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.872570 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:27.879463 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:27.908321 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:10:27.955458 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.001303 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:28.005263 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.018517 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:28.085985 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.104892 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.239344 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.390487 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:28.394222 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:28.429646 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.480676 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.567155 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:28.678215 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:28.678959 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:28.688889 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:28.854481 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:28.995275 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:29.099861 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:29.160552 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:29.400574 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:29.446480 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.482511 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.515712 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.528947 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:10:29.549580 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.625430 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.657461 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.796830 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.862169 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.871582 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:29.893337 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:29.965323 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.021198 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.028348 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.265334 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.391955 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.406441 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.448284 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:30.461740 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.464046 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.634560 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.752200 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.833442 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.879449 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:30.887351 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:30.983194 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:31.146339 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:31.282524 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:31.298168 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:31.555580 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:31.678834 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:31.682148 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:31.704656 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:31.859872 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:31.904105 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.124011 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:32.152351 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.175729 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:10:32.185882 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.220352 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.240219 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.319258 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.331377 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.399946 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.526265 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.532205 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.594802 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:32.645493 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:32.646262 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.727360 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.877791 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.956454 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:32.970994 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:10:32.989655 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:33.010002 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:33.020498 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:33.146957 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:33.179111 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:33.314517 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:33.331857 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:33.360793 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:33.429204 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:33.512630 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:33.546128 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:33.616741 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:33.628080 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:33.901643 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:33.970917 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:10:34.030200 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.056916 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:34.104555 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:34.140280 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:34.459766 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.492009 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.615399 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:10:34.637620 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.641736 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.641796 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.672241 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:34.693912 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.753773 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.764539 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.891192 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.933914 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.936739 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:34.939968 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.082183 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.100015 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:35.194821 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.380245 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.434458 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.604174 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.649991 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.706321 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:35.859926 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.870764 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:35.891100 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:35.893708 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:36.284670 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:36.405514 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:36.480654 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:36.548370 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:36.728044 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:36.733220 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:36.765271 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:36.801823 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:10:36.998435 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.049271 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.071895 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.188509 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.275571 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.298460 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:37.323071 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.329177 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.335572 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.475450 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.558501 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.559943 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.600608 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.724275 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:37.764807 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.800246 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:37.831657 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.988093 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:37.989264 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:38.056238 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:38.058137 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:10:38.058715 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:38.155668 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:38.209578 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:38.216686 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:38.395094 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:38.422345 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:38.499871 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:38.508929 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:38.663542 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:38.753562 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:38.950650 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.083676 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:39.084282 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.177430 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:39.195874 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:39.264873 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.322829 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:39.368428 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:10:39.370097 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.415221 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.583671 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.603030 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.630604 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:39.704236 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.808398 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.870615 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.875610 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.875700 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.922228 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:39.942950 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.128582 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:40.221940 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.386361 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.462716 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.506268 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:40.526134 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.577175 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.580129 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.655077 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:40.794348 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.878576 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:40.965921 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:41.000109 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:41.159819 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:41.452914 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:41.553161 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:41.576676 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:41.752336 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:41.789456 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:41.818382 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:41.877360 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:41.911722 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:10:42.025921 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.060737 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.084324 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:42.100946 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.169382 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.235536 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.280678 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.493132 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.524776 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:42.551611 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.600080 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.695509 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.813514 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:10:42.841557 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.876096 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:42.946974 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:43.062341 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:43.071130 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:43.079569 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:43.101557 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:10:43.161314 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:10:43.200186 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:43.200850 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:43.397305 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:43.438684 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:43.559554 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:43.613883 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:43.635955 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:43.675307 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:43.828393 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:43.930025 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:44.158921 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.176712 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:44.177378 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:44.203753 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.212777 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:44.386245 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.395321 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:10:44.428701 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.499897 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.585202 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.594041 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.631806 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:44.663667 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.702648 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.718057 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.852545 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:44.976676 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.007391 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.079560 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:45.206228 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.210480 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.256311 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.472661 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.655983 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:45.745206 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.818986 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:45.897190 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:45.970124 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:46.052147 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:46.084203 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:46.094439 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:46.163523 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:46.206470 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:46.444748 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:46.518819 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:46.689605 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:46.696907 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:46.782651 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:46.886152 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:46.921547 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:46.975684 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.033299 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:10:47.036644 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.146204 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.179535 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.213490 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.230983 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:47.273936 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.319875 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.397452 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.676820 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:47.763324 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.832432 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:47.910784 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:47.950008 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.007524 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 14:10:48.008270 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:48.084812 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:48.119493 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:10:48.157480 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.209414 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:48.210107 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:48.213165 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.253091 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:48.403033 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.550805 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:48.565477 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:48.566046 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:48.592141 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.615098 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.618643 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.719313 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:48.900684 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:48.955204 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:49.081120 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:49.081763 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.113834 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:49.153710 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.193863 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:49.249862 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.342607 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:49.361315 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.417068 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.493052 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.507329 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:49.538024 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.545015 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.634879 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.636020 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.669379 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.830677 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:49.842442 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:49.878929 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:10:50.165560 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:50.272627 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:50.286412 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:50.337507 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:50.575560 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:50.691908 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:50.740447 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:50.858599 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:50.949409 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:50.964785 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:51.042412 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.073015 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:51.118203 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:51.146586 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:51.431973 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:51.506444 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.519953 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.553766 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.626565 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.644968 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:51.759374 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.872427 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.889801 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:51.955902 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:51.983208 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.013108 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.110102 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.180379 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.230017 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:10:52.301322 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.407908 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:52.464066 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.748973 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.841066 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:52.846308 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.854876 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:52.857995 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:52.920095 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:53.097628 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.156762 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.186271 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:10:53.203944 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.272616 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:53.320924 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:10:53.354171 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.464411 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:53.495496 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.577888 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:10:53.578581 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:53.596994 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.724720 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:53.974632 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.018244 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:54.023485 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.044227 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:54.124378 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:54.136020 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.184615 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.220472 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:54.259353 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.385484 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.427124 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:54.474858 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.546321 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.578464 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.598623 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.612880 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.623572 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.719470 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:10:54.811177 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:54.959434 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:54.977048 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.148182 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:55.247087 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.466259 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:55.520437 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.604070 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.702662 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.761638 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.814131 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.928555 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:55.941090 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:55.975589 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:56.104564 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:56.114879 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:56.323565 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:56.448635 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.467758 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.583750 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:56.591977 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.759764 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.767144 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.794718 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:10:56.917306 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.935417 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:56.997210 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.039391 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:57.046928 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.134714 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.139313 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.143486 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.332168 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.498940 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:57.514562 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:57.552706 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.796962 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.843974 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:57.941123 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:57.980036 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.047852 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.054312 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:58.184593 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.301405 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:58.343776 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:10:58.451985 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.453075 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.458190 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:58.533150 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:58.664170 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.666155 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:58.779814 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:10:58.813215 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.827581 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:58.892246 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.136629 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.138613 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.170604 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:59.172935 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.227541 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.283231 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:10:59.294405 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:10:59.418561 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.490945 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.519963 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.537319 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:10:59.600992 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.624214 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.675616 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:10:59.778009 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.831297 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:10:59.868338 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:10:59.905118 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.034710 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.051502 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:00.256649 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.348675 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.375454 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.583039 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.613662 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:00.624320 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:00.721379 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:00.767060 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:01.032195 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:01.053644 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:01.116367 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:01.179725 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:01.236119 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:01.363585 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:01.578359 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:01.578612 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:01.640717 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:01.658969 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:01.793747 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:01.913025 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:01.987252 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.045397 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.069384 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.136548 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:02.168272 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:02.214020 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.368268 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.410141 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:02.426771 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.517127 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:02.556499 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.652572 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.742971 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.838296 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.946069 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:02.954349 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:11:02.967228 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.032898 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.033483 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:03.309001 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.364168 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:03.369717 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:03.552618 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:03.573498 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:03.576084 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.576622 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:03.608865 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.660969 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.778095 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:03.868592 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:03.985244 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.038859 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:04.079124 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.126508 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.209215 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.257766 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.298914 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:04.309253 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.437732 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:04.511976 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:04.514984 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:11:04.535158 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.662285 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.694375 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.788836 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.794903 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.925812 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.981843 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:04.994978 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:05.162154 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:05.224542 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.240824 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.409328 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.458308 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.480937 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:05.504766 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.542557 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.952507 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:05.968950 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:06.020025 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:06.035045 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:06.035701 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.202132 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:06.343263 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:06.461381 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.494158 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.497352 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.580175 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.612869 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:06.630146 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.845208 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:06.955009 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.013639 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:07.031438 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:11:07.112792 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.153076 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.247226 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.248121 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.545012 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.627694 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.628904 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:07.649390 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.711526 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:07.800041 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.907979 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.917398 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.963572 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:07.969913 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:08.034225 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:11:08.150192 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:08.428526 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:08.502593 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:11:08.597846 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:08.616219 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:08.632035 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:08.657094 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:11:08.664204 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:08.684427 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:08.723212 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:08.739075 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:08.750592 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:08.889475 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:08.973392 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.039820 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.059195 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:09.169126 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.289168 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:09.322013 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:09.445114 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.459194 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:09.605101 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:11:09.623204 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.673684 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.770941 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.805215 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.875062 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.875726 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:09.895910 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:09.915818 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.061017 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:10.188756 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.283561 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.301508 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.332054 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:10.516438 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.636137 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.671403 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.886673 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:10.961385 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:10.963346 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:10.969101 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:11.041872 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:11.091542 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:11.247087 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:11.283004 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:11.413280 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:11.457505 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:11.518523 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:11.519310 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:11.571336 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:11.984905 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:11.998097 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:12.113729 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.117357 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:11:12.174842 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.178483 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.200798 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.313900 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.353118 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.491655 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:12.565330 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.597856 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:12.654012 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.728830 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.944005 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:12.945744 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:13.030625 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:13.081244 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:13.124579 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:13.146035 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:13.209235 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:13.402976 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:13.459695 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:13.469198 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:13.535637 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:13.537339 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:13.619566 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:13.638226 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:13.688982 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:11:13.824990 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:13.930374 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:13.983675 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:14.010957 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:14.024739 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.050251 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:14.076189 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.255115 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.278894 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:14.390873 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:14.401181 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:14.454483 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.464195 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.520143 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.531062 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:11:14.623263 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:14.701324 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.854784 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:14.854993 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:15.069772 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.080474 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:15.160598 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.173398 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.383472 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.403573 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.411417 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:15.463654 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.536057 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:15.568502 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:15.744375 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:11:15.759325 12434 maintenance_manager.cc:419] P bd843b84065446c09f71ac8534f21a8a: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033368
W20240501 14:11:15.771508 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:11:15.796662 12433 maintenance_manager.cc:643] P bd843b84065446c09f71ac8534f21a8a: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.033s sys 0.000s Metrics: {"bytes_written":114460,"cfile_init":1,"dirs.queue_time_us":203,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1255,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":999,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":323,"thread_start_us":187,"threads_started":2,"wal-append.queue_time_us":150}
I20240501 14:11:15.797547 12434 maintenance_manager.cc:419] P bd843b84065446c09f71ac8534f21a8a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 41420 bytes on disk
I20240501 14:11:15.798063 12433 maintenance_manager.cc:643] P bd843b84065446c09f71ac8534f21a8a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 14:11:15.828518 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:15.967052 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:15.968359 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:16.026147 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:16.197252 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:16.199101 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:16.315714 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:16.341071 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:16.408576 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:16.672442 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:16.714610 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:16.751449 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:16.784176 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:16.874930 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.018656 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.050982 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.088241 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.126052 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:11:17.352592 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:17.442481 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.647653 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.666889 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:17.837340 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.859292 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.864065 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:17.914158 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.952777 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:17.996224 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:18.153264 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:11:18.210688 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:18.212579 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:18.227546 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:18.232249 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:18.305474 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:18.325192 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:18.478430 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:18.534433 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:18.612712 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:18.681041 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:18.717449 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:11:18.718017 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:18.729722 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:18.853894 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:18.998346 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:19.199733 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:19.200639 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:19.326606 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:19.354377 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:19.356832 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:19.404189 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:19.421041 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:19.465523 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:19.479607 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:19.480201 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:11:19.565162 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:19.629686 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:11:19.657892 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:19.950085 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.009733 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:20.056236 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:20.219236 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.288259 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.346735 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.392630 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.491382 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:20.530161 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.623098 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.718324 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:20.802755 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.857035 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:20.886682 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:20.900687 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:20.967265 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:21.002122 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:21.023929 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:21.142307 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:21.160750 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:21.165756 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:21.263890 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:21.347709 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:21.556423 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(66436017fd044f0b919629933ea56e9c): perf score=0.033336
W20240501 14:11:21.571969 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:11:21.584566 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(66436017fd044f0b919629933ea56e9c) complete. Timing: real 0.028s user 0.020s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":152,"dirs.queue_time_us":6031,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":404,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":249,"threads_started":3,"wal-append.queue_time_us":471}
I20240501 14:11:21.585521 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(a5957da39df64253a2b8d7f388308719): perf score=0.033341
I20240501 14:11:21.587234 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(66436017fd044f0b919629933ea56e9c): 4834 bytes on disk
I20240501 14:11:21.587812 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(66436017fd044f0b919629933ea56e9c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:11:21.640838 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(a5957da39df64253a2b8d7f388308719) complete. Timing: real 0.055s user 0.030s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":170,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1471,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":643,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":265,"threads_started":3,"wal-append.queue_time_us":797}
I20240501 14:11:21.643124 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(a5957da39df64253a2b8d7f388308719): 4921 bytes on disk
I20240501 14:11:21.644030 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(a5957da39df64253a2b8d7f388308719) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 14:11:21.703884 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:21.711879 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(d64234effb3446bba1ff82bdf37acaa7): perf score=0.033377
I20240501 14:11:21.772315 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(a5957da39df64253a2b8d7f388308719): perf score=0.033397
I20240501 14:11:21.808145 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(d64234effb3446bba1ff82bdf37acaa7) complete. Timing: real 0.096s user 0.034s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":180,"dirs.queue_time_us":11635,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":2444,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":239,"threads_started":3,"wal-append.queue_time_us":1615}
I20240501 14:11:21.810693 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(d64234effb3446bba1ff82bdf37acaa7): 4581 bytes on disk
I20240501 14:11:21.822701 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(d64234effb3446bba1ff82bdf37acaa7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240501 14:11:21.827265 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:21.834882 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:21.864504 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(a5957da39df64253a2b8d7f388308719) complete. Timing: real 0.092s user 0.027s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":33354,"dirs.queue_time_us":15686,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":3691,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":825,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":304,"threads_started":3,"wal-append.queue_time_us":4917}
I20240501 14:11:21.867165 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(66436017fd044f0b919629933ea56e9c): perf score=0.033419
W20240501 14:11:21.870645 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:21.908562 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(66436017fd044f0b919629933ea56e9c) complete. Timing: real 0.041s user 0.015s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":3186,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11422,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":120,"threads_started":1,"wal-append.queue_time_us":10700}
I20240501 14:11:21.910634 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(a5957da39df64253a2b8d7f388308719): 4921 bytes on disk
I20240501 14:11:21.911346 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(a5957da39df64253a2b8d7f388308719) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:11:21.913120 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(66436017fd044f0b919629933ea56e9c): 4834 bytes on disk
I20240501 14:11:21.913499 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(66436017fd044f0b919629933ea56e9c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240501 14:11:21.965780 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:22.145314 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:22.217285 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:22.234196 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:11:22.236681 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:22.259302 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:22.617338 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:22.653458 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(eb336cffd30b447b880af97da6882d27): perf score=0.033337
W20240501 14:11:22.665987 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:22.666901 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(eb336cffd30b447b880af97da6882d27) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1405,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":258}
I20240501 14:11:22.668823 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(eb336cffd30b447b880af97da6882d27): 4581 bytes on disk
I20240501 14:11:22.669713 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(e13e55ae52a949bc893de94082130157): perf score=0.033346
I20240501 14:11:22.670286 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(eb336cffd30b447b880af97da6882d27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240501 14:11:22.674480 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:22.687752 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(e13e55ae52a949bc893de94082130157) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1579,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":112}
I20240501 14:11:22.689392 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(e13e55ae52a949bc893de94082130157): 4834 bytes on disk
I20240501 14:11:22.689705 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(e13e55ae52a949bc893de94082130157) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240501 14:11:22.827852 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:22.831187 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(7693e3e204634676a758726bb6d3d2ce): perf score=0.033389
I20240501 14:11:22.845782 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(7693e3e204634676a758726bb6d3d2ce) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1603,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":387,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:22.847302 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(e13e55ae52a949bc893de94082130157): perf score=0.033392
I20240501 14:11:22.847888 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(eb336cffd30b447b880af97da6882d27): perf score=0.033392
I20240501 14:11:22.861351 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(eb336cffd30b447b880af97da6882d27) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1736,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":382,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":164}
I20240501 14:11:22.863461 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(eb336cffd30b447b880af97da6882d27): 4581 bytes on disk
I20240501 14:11:22.863828 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(eb336cffd30b447b880af97da6882d27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 14:11:22.865293 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(e13e55ae52a949bc893de94082130157) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":2435,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":487,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":137}
I20240501 14:11:22.867213 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(7693e3e204634676a758726bb6d3d2ce): 4921 bytes on disk
I20240501 14:11:22.871038 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(7693e3e204634676a758726bb6d3d2ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:11:22.872839 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(e13e55ae52a949bc893de94082130157): 4834 bytes on disk
I20240501 14:11:22.873255 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(e13e55ae52a949bc893de94082130157) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240501 14:11:22.919940 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.085364 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.109470 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.206228 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.209833 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.210078 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:23.232080 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:23.283418 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:23.290632 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:23.326402 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.450343 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.458879 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.479243 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:23.565151 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:23.592245 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:23.612998 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:23.613526 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:23.764983 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:11:23.788983 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 14:11:23.871632 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(7c778ae9911740c2b0cb42dfbc694bb1): perf score=0.033366
I20240501 14:11:23.881600 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(727a51852ac2428eb23f7690fec56a16): perf score=0.033369
W20240501 14:11:23.886081 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:23.896029 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(727a51852ac2428eb23f7690fec56a16) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":460,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":121}
I20240501 14:11:23.896366 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(7c778ae9911740c2b0cb42dfbc694bb1) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":239,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1545,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":160}
I20240501 14:11:23.898087 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(727a51852ac2428eb23f7690fec56a16): 4921 bytes on disk
I20240501 14:11:23.898187 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(7c778ae9911740c2b0cb42dfbc694bb1): 4834 bytes on disk
I20240501 14:11:23.898475 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(727a51852ac2428eb23f7690fec56a16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 14:11:23.898531 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(7c778ae9911740c2b0cb42dfbc694bb1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:11:23.900394 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(7c778ae9911740c2b0cb42dfbc694bb1): perf score=0.033372
W20240501 14:11:23.907403 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:11:23.912770 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(7c778ae9911740c2b0cb42dfbc694bb1) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":396,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":155}
I20240501 14:11:23.914288 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(7c778ae9911740c2b0cb42dfbc694bb1): 4834 bytes on disk
I20240501 14:11:23.914654 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(7c778ae9911740c2b0cb42dfbc694bb1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:11:23.930956 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(63a76e89d58d48ba9d1b3b98ef6c172d): perf score=0.033381
I20240501 14:11:23.944952 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(63a76e89d58d48ba9d1b3b98ef6c172d) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1397,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":456,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":169}
I20240501 14:11:23.946733 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(63a76e89d58d48ba9d1b3b98ef6c172d): 4581 bytes on disk
I20240501 14:11:23.947225 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(63a76e89d58d48ba9d1b3b98ef6c172d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 14:11:23.949522 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(727a51852ac2428eb23f7690fec56a16): perf score=0.033386
I20240501 14:11:23.964908 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(727a51852ac2428eb23f7690fec56a16) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":2938,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":569,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":117,"threads_started":2,"wal-append.queue_time_us":123}
I20240501 14:11:23.967074 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(727a51852ac2428eb23f7690fec56a16): 4921 bytes on disk
I20240501 14:11:23.967435 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(727a51852ac2428eb23f7690fec56a16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 14:11:23.976105 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.104030 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.217535 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:24.233042 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.379520 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:24.445255 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:24.448696 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.542347 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.650223 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.669207 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:24.706167 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:24.727370 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:11:24.799913 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:24.800633 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:24.905953 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(515d8f8e9d744aa4b2c8a4090c2f72e8): perf score=0.033360
I20240501 14:11:24.920892 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(515d8f8e9d744aa4b2c8a4090c2f72e8) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":2071,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":120,"threads_started":2,"wal-append.queue_time_us":587}
I20240501 14:11:24.922828 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(515d8f8e9d744aa4b2c8a4090c2f72e8): 4921 bytes on disk
I20240501 14:11:24.922959 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(515d8f8e9d744aa4b2c8a4090c2f72e8): perf score=0.033365
I20240501 14:11:24.923218 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(515d8f8e9d744aa4b2c8a4090c2f72e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 14:11:24.936597 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(515d8f8e9d744aa4b2c8a4090c2f72e8) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1472,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":149}
I20240501 14:11:24.938149 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(515d8f8e9d744aa4b2c8a4090c2f72e8): 4921 bytes on disk
I20240501 14:11:24.938474 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(515d8f8e9d744aa4b2c8a4090c2f72e8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 14:11:24.939908 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(5648b7e0a1e744cea470feb7911d3d2c): perf score=0.033368
I20240501 14:11:24.952666 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(5648b7e0a1e744cea470feb7911d3d2c) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":1239,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":123}
I20240501 14:11:24.954695 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(5648b7e0a1e744cea470feb7911d3d2c): 4581 bytes on disk
I20240501 14:11:24.955111 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(5648b7e0a1e744cea470feb7911d3d2c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 14:11:24.955401 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(5648b7e0a1e744cea470feb7911d3d2c): perf score=0.033371
I20240501 14:11:24.968071 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(5648b7e0a1e744cea470feb7911d3d2c) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":460,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":146}
I20240501 14:11:24.969488 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(5648b7e0a1e744cea470feb7911d3d2c): 4581 bytes on disk
I20240501 14:11:24.969826 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(5648b7e0a1e744cea470feb7911d3d2c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:11:24.971293 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(16324878552c4b6dbf10e8d5f73a0436): perf score=0.033374
I20240501 14:11:24.976498 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(16324878552c4b6dbf10e8d5f73a0436): perf score=0.033377
I20240501 14:11:24.985244 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(16324878552c4b6dbf10e8d5f73a0436) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":507,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":174}
I20240501 14:11:24.986914 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(16324878552c4b6dbf10e8d5f73a0436): 4834 bytes on disk
I20240501 14:11:24.987298 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(16324878552c4b6dbf10e8d5f73a0436) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:11:24.992388 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(16324878552c4b6dbf10e8d5f73a0436) complete. Timing: real 0.016s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":217,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1319,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:24.994318 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(16324878552c4b6dbf10e8d5f73a0436): 4834 bytes on disk
I20240501 14:11:24.994637 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(16324878552c4b6dbf10e8d5f73a0436) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 14:11:25.032694 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.266070 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:25.313568 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.327049 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:25.394385 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.424130 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.617257 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:25.625527 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.635713 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.647136 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.670235 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:25.787775 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:25.811403 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:25.879985 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:25.957692 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:25.963225 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(8147fae0de98429ab6554c7e7f5bf8ee): perf score=0.033335
W20240501 14:11:25.968075 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:11:25.975730 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(8147fae0de98429ab6554c7e7f5bf8ee) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":122}
I20240501 14:11:25.977258 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(8147fae0de98429ab6554c7e7f5bf8ee): 4921 bytes on disk
I20240501 14:11:25.977622 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(8147fae0de98429ab6554c7e7f5bf8ee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 14:11:25.979068 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(d48da29c058b4db3a5b45b25755be1cf): perf score=0.033336
I20240501 14:11:25.991664 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(d48da29c058b4db3a5b45b25755be1cf) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1535,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":125}
I20240501 14:11:25.993122 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(d48da29c058b4db3a5b45b25755be1cf): 4834 bytes on disk
I20240501 14:11:25.993469 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(d48da29c058b4db3a5b45b25755be1cf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 14:11:25.995364 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(8147fae0de98429ab6554c7e7f5bf8ee): perf score=0.033343
I20240501 14:11:26.002452 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(d48da29c058b4db3a5b45b25755be1cf): perf score=0.033344
I20240501 14:11:26.008955 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(8147fae0de98429ab6554c7e7f5bf8ee) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":530,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":139}
I20240501 14:11:26.011054 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(68b04992f1f94b36b01dfde24a42318d): perf score=0.033345
I20240501 14:11:26.015841 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(d48da29c058b4db3a5b45b25755be1cf) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1081,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":130}
I20240501 14:11:26.017779 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(d48da29c058b4db3a5b45b25755be1cf): 4834 bytes on disk
I20240501 14:11:26.018085 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(d48da29c058b4db3a5b45b25755be1cf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 14:11:26.024245 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(68b04992f1f94b36b01dfde24a42318d) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":981,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":175}
I20240501 14:11:26.025640 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(8147fae0de98429ab6554c7e7f5bf8ee): 4921 bytes on disk
I20240501 14:11:26.025977 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(8147fae0de98429ab6554c7e7f5bf8ee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240501 14:11:26.027493 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 14:11:26.027803 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(68b04992f1f94b36b01dfde24a42318d): 4581 bytes on disk
I20240501 14:11:26.028636 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(68b04992f1f94b36b01dfde24a42318d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 14:11:26.179279 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:26.181741 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:11:26.182617 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(d48da29c058b4db3a5b45b25755be1cf): perf score=0.033394
I20240501 14:11:26.195235 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(d48da29c058b4db3a5b45b25755be1cf) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":483,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":143}
I20240501 14:11:26.197070 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(d48da29c058b4db3a5b45b25755be1cf): 4834 bytes on disk
I20240501 14:11:26.197412 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(d48da29c058b4db3a5b45b25755be1cf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 14:11:26.340952 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:26.434880 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:26.443063 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:26.487857 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:26.709517 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:26.862804 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:26.876956 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:11:26.953096 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(49f2b0e67d3c43d584ac7583b98caeb9): perf score=0.033359
I20240501 14:11:26.967031 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(49f2b0e67d3c43d584ac7583b98caeb9) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":3859,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":168,"threads_started":2,"wal-append.queue_time_us":155}
W20240501 14:11:26.968657 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:26.969071 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(01d9bd5e798f4bf2a5dd1eedcb4676cc): perf score=0.033361
I20240501 14:11:26.984468 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(01d9bd5e798f4bf2a5dd1eedcb4676cc) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1997,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":497,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":181}
I20240501 14:11:26.986057 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(01d9bd5e798f4bf2a5dd1eedcb4676cc): 4921 bytes on disk
I20240501 14:11:26.986411 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(01d9bd5e798f4bf2a5dd1eedcb4676cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 14:11:26.987663 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(49f2b0e67d3c43d584ac7583b98caeb9): 4581 bytes on disk
I20240501 14:11:26.987982 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(49f2b0e67d3c43d584ac7583b98caeb9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 14:11:27.001967 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(bd5489eab5b04751bc8b3e5fb9e8cf73): perf score=0.033372
I20240501 14:11:27.015211 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(bd5489eab5b04751bc8b3e5fb9e8cf73) complete. Timing: real 0.013s user 0.012s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1320,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":506,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":131}
I20240501 14:11:27.016954 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(bd5489eab5b04751bc8b3e5fb9e8cf73): 4832 bytes on disk
I20240501 14:11:27.017521 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(bd5489eab5b04751bc8b3e5fb9e8cf73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 14:11:27.025751 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(01d9bd5e798f4bf2a5dd1eedcb4676cc): perf score=0.033378
I20240501 14:11:27.036006 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(49f2b0e67d3c43d584ac7583b98caeb9): perf score=0.033378
I20240501 14:11:27.039039 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(01d9bd5e798f4bf2a5dd1eedcb4676cc) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1268,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":182,"threads_started":2,"wal-append.queue_time_us":173}
I20240501 14:11:27.041163 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(01d9bd5e798f4bf2a5dd1eedcb4676cc): 4921 bytes on disk
I20240501 14:11:27.041481 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(01d9bd5e798f4bf2a5dd1eedcb4676cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 14:11:27.050372 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(49f2b0e67d3c43d584ac7583b98caeb9) complete. Timing: real 0.014s user 0.010s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1270,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":415,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":124}
I20240501 14:11:27.052062 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(49f2b0e67d3c43d584ac7583b98caeb9): 4581 bytes on disk
I20240501 14:11:27.052397 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(49f2b0e67d3c43d584ac7583b98caeb9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 14:11:27.073608 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.120949 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.132592 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:11:27.145396 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.356307 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:27.423769 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.764487 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.820117 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:27.855458 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:27.862805 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.894810 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:27.962661 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:11:27.995460 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(40b29fb993f34e99ad3747a2ec7d6c4f): perf score=0.033373
I20240501 14:11:28.010860 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(40b29fb993f34e99ad3747a2ec7d6c4f) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3343,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1668,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":700}
I20240501 14:11:28.012363 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(40b29fb993f34e99ad3747a2ec7d6c4f): 4833 bytes on disk
I20240501 14:11:28.012782 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(40b29fb993f34e99ad3747a2ec7d6c4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:11:28.025370 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(6628290de33e4aba8ccd1651d1d054dd): perf score=0.033380
I20240501 14:11:28.039430 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(6628290de33e4aba8ccd1651d1d054dd) complete. Timing: real 0.014s user 0.011s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1340,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":501,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":118}
I20240501 14:11:28.041435 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(6628290de33e4aba8ccd1651d1d054dd): 4580 bytes on disk
I20240501 14:11:28.042330 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(6628290de33e4aba8ccd1651d1d054dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 14:11:28.049098 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(abac52c109174021ac11360b0d22ccfa): perf score=0.033390
I20240501 14:11:28.060186 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(abac52c109174021ac11360b0d22ccfa): perf score=0.033392
I20240501 14:11:28.074807 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(abac52c109174021ac11360b0d22ccfa) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":3671,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":10470}
I20240501 14:11:28.074864 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(abac52c109174021ac11360b0d22ccfa) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1361,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":146}
I20240501 14:11:28.076799 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(6628290de33e4aba8ccd1651d1d054dd): perf score=0.033395
I20240501 14:11:28.076853 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(abac52c109174021ac11360b0d22ccfa): 4922 bytes on disk
I20240501 14:11:28.077235 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(abac52c109174021ac11360b0d22ccfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 14:11:28.078500 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(40b29fb993f34e99ad3747a2ec7d6c4f): perf score=0.033397
I20240501 14:11:28.091374 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(6628290de33e4aba8ccd1651d1d054dd) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1154,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":146}
I20240501 14:11:28.093042 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(40b29fb993f34e99ad3747a2ec7d6c4f) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":965,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":430,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1792,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":925}
I20240501 14:11:28.093501 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(abac52c109174021ac11360b0d22ccfa): 4922 bytes on disk
I20240501 14:11:28.094285 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(abac52c109174021ac11360b0d22ccfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240501 14:11:28.095083 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(40b29fb993f34e99ad3747a2ec7d6c4f): 4833 bytes on disk
I20240501 14:11:28.095474 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(40b29fb993f34e99ad3747a2ec7d6c4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:11:28.096061 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(6628290de33e4aba8ccd1651d1d054dd): 4580 bytes on disk
I20240501 14:11:28.096417 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(6628290de33e4aba8ccd1651d1d054dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 14:11:28.100028 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.120620 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:28.175551 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.200558 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.250945 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:11:28.383981 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:28.384871 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.419190 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.470094 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:28.477064 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:28.507567 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.630352 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:28.638337 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:28.656510 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:28.815217 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:11:28.859616 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:11:28.881361 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:28.983407 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:29.010421 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:11:29.020481 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(d178c3291efc44369e3adb921ccc6517): perf score=0.033379
I20240501 14:11:29.032784 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(d178c3291efc44369e3adb921ccc6517) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1418,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":144}
I20240501 14:11:29.034317 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(d178c3291efc44369e3adb921ccc6517): 4581 bytes on disk
I20240501 14:11:29.034646 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(d178c3291efc44369e3adb921ccc6517) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 14:11:29.051084 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(218f8a4fae324dc7bc535e6a18addb61): perf score=0.033387
I20240501 14:11:29.068948 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(218f8a4fae324dc7bc535e6a18addb61) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2734,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1832,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":506,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":137}
I20240501 14:11:29.070380 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(218f8a4fae324dc7bc535e6a18addb61): 4921 bytes on disk
I20240501 14:11:29.070791 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(218f8a4fae324dc7bc535e6a18addb61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:11:29.103619 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(218f8a4fae324dc7bc535e6a18addb61): perf score=0.033403
I20240501 14:11:29.105978 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(56197e787ff14e85a07060f23e612498): perf score=0.033400
I20240501 14:11:29.119444 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(218f8a4fae324dc7bc535e6a18addb61) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":129,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1225,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":517,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":112,"threads_started":2,"wal-append.queue_time_us":118}
I20240501 14:11:29.120764 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(56197e787ff14e85a07060f23e612498) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1079,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":184}
I20240501 14:11:29.121474 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(56197e787ff14e85a07060f23e612498): perf score=0.033407
I20240501 14:11:29.122601 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(56197e787ff14e85a07060f23e612498): 4834 bytes on disk
I20240501 14:11:29.123158 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(56197e787ff14e85a07060f23e612498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 14:11:29.135500 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(56197e787ff14e85a07060f23e612498) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1267,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":446,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":186}
I20240501 14:11:29.137244 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(218f8a4fae324dc7bc535e6a18addb61): 4921 bytes on disk
I20240501 14:11:29.138115 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(218f8a4fae324dc7bc535e6a18addb61) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":397,"lbm_reads_lt_1ms":4}
I20240501 14:11:29.139761 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(56197e787ff14e85a07060f23e612498): 4834 bytes on disk
I20240501 14:11:29.140040 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(56197e787ff14e85a07060f23e612498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 14:11:29.241644 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:29.395278 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:29.397392 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:29.469496 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:29.484870 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:29.515264 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:11:29.625217 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:29.667503 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:11:29.701902 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:29.738437 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:29.788496 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:29.846426 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:29.941982 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.040722 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:11:30.043223 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(d01550698e8449b081a339f6ba30591c): perf score=0.033347
I20240501 14:11:30.056347 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(d01550698e8449b081a339f6ba30591c) complete. Timing: real 0.013s user 0.008s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1347,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":390,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":137}
I20240501 14:11:30.058255 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(d01550698e8449b081a339f6ba30591c): 4834 bytes on disk
I20240501 14:11:30.060053 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(d01550698e8449b081a339f6ba30591c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 14:11:30.078603 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(d01550698e8449b081a339f6ba30591c): perf score=0.033357
I20240501 14:11:30.090137 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(d01550698e8449b081a339f6ba30591c) complete. Timing: real 0.011s user 0.003s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":191,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1361,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":428,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":119}
I20240501 14:11:30.091918 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(a0e8e476bc46420d992a43bf3b4c73fd): perf score=0.033360
I20240501 14:11:30.104033 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(a0e8e476bc46420d992a43bf3b4c73fd) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2578,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":419,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":165}
I20240501 14:11:30.105746 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(d01550698e8449b081a339f6ba30591c): 4834 bytes on disk
I20240501 14:11:30.106137 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(d01550698e8449b081a339f6ba30591c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 14:11:30.107780 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(a0e8e476bc46420d992a43bf3b4c73fd): 4581 bytes on disk
I20240501 14:11:30.108088 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(a0e8e476bc46420d992a43bf3b4c73fd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 14:11:30.130708 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(a0e8e476bc46420d992a43bf3b4c73fd): perf score=0.033375
I20240501 14:11:30.142454 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(a0e8e476bc46420d992a43bf3b4c73fd) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1583,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":454,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":116}
I20240501 14:11:30.143862 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(1f1c8ff4149c42aa9517af442c38060c): perf score=0.033375
I20240501 14:11:30.147400 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(1f1c8ff4149c42aa9517af442c38060c): perf score=0.033380
I20240501 14:11:30.155359 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(1f1c8ff4149c42aa9517af442c38060c) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1264,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":133}
I20240501 14:11:30.157079 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(1f1c8ff4149c42aa9517af442c38060c): 4921 bytes on disk
I20240501 14:11:30.157418 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(1f1c8ff4149c42aa9517af442c38060c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
I20240501 14:11:30.159008 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(a0e8e476bc46420d992a43bf3b4c73fd): 4581 bytes on disk
I20240501 14:11:30.159293 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(a0e8e476bc46420d992a43bf3b4c73fd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 14:11:30.163731 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(1f1c8ff4149c42aa9517af442c38060c) complete. Timing: real 0.016s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":943,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1137,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":129}
I20240501 14:11:30.165177 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(1f1c8ff4149c42aa9517af442c38060c): 4921 bytes on disk
I20240501 14:11:30.165509 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(1f1c8ff4149c42aa9517af442c38060c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 14:11:30.217047 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.294113 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.307920 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:30.409354 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:30.413635 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:30.516810 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.532630 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:30.611984 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.711189 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.788564 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:30.850706 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:30.895538 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:30.907809 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:30.950373 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:31.000281 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:31.103427 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:31.113188 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:31.152935 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:31.174979 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:31.197042 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:31.297142 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:31.319202 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling FlushMRSOp(0c3018b4356d462a9f2dd5511392661c): perf score=0.033363
I20240501 14:11:31.332842 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: FlushMRSOp(0c3018b4356d462a9f2dd5511392661c) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":135}
I20240501 14:11:31.334607 12634 maintenance_manager.cc:419] P 6a3cc0660eed427eaee38164a8c1d4c3: Scheduling UndoDeltaBlockGCOp(0c3018b4356d462a9f2dd5511392661c): 4581 bytes on disk
I20240501 14:11:31.334990 12524 maintenance_manager.cc:643] P 6a3cc0660eed427eaee38164a8c1d4c3: UndoDeltaBlockGCOp(0c3018b4356d462a9f2dd5511392661c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:11:31.367178 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(0c3018b4356d462a9f2dd5511392661c): perf score=0.033376
I20240501 14:11:31.381424 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(0c3018b4356d462a9f2dd5511392661c) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1381,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":150}
I20240501 14:11:31.384459 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling FlushMRSOp(444c1a9d0e6943dc880161ae320673b6): perf score=0.033380
I20240501 14:11:31.401237 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: FlushMRSOp(444c1a9d0e6943dc880161ae320673b6) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1547,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":471,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":172}
I20240501 14:11:31.402633 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(444c1a9d0e6943dc880161ae320673b6): 4921 bytes on disk
I20240501 14:11:31.403028 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(444c1a9d0e6943dc880161ae320673b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 14:11:31.404625 12896 maintenance_manager.cc:419] P 4902d2609e6d44d089590a42e70356b2: Scheduling UndoDeltaBlockGCOp(0c3018b4356d462a9f2dd5511392661c): 4581 bytes on disk
I20240501 14:11:31.405000 12786 maintenance_manager.cc:643] P 4902d2609e6d44d089590a42e70356b2: UndoDeltaBlockGCOp(0c3018b4356d462a9f2dd5511392661c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:11:31.419301 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling FlushMRSOp(25b5c40793b348bbb0e5251c8d553e30): perf score=0.033391
I20240501 14:11:31.425753 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(444c1a9d0e6943dc880161ae320673b6): perf score=0.033394
I20240501 14:11:31.434785 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: FlushMRSOp(25b5c40793b348bbb0e5251c8d553e30) complete. Timing: real 0.015s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3025,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":248}
I20240501 14:11:31.436393 13027 maintenance_manager.cc:419] P a6e94562bee149c2ace25672f0f9bf50: Scheduling UndoDeltaBlockGCOp(25b5c40793b348bbb0e5251c8d553e30): 4834 bytes on disk
I20240501 14:11:31.436725 12917 maintenance_manager.cc:643] P a6e94562bee149c2ace25672f0f9bf50: UndoDeltaBlockGCOp(25b5c40793b348bbb0e5251c8d553e30) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240501 14:11:31.441283 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(444c1a9d0e6943dc880161ae320673b6) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1227,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":506,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":117}
I20240501 14:11:31.442965 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling FlushMRSOp(25b5c40793b348bbb0e5251c8d553e30): perf score=0.033396
I20240501 14:11:31.460703 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: FlushMRSOp(25b5c40793b348bbb0e5251c8d553e30) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1550,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":935}
I20240501 14:11:31.462492 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(444c1a9d0e6943dc880161ae320673b6): 4921 bytes on disk
I20240501 14:11:31.462883 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(444c1a9d0e6943dc880161ae320673b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 14:11:31.464157 12765 maintenance_manager.cc:419] P e6ab42bc38694f7f9fc8bfcc0a1483b7: Scheduling UndoDeltaBlockGCOp(25b5c40793b348bbb0e5251c8d553e30): 4834 bytes on disk
I20240501 14:11:31.464490 12655 maintenance_manager.cc:643] P e6ab42bc38694f7f9fc8bfcc0a1483b7: UndoDeltaBlockGCOp(25b5c40793b348bbb0e5251c8d553e30) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 14:11:31.505455 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:31.623186 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:31.695401 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:31.868424 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:31.931648 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:32.036298 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:32.128628 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:32.218605 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:32.231247 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:32.322221 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:32.425525 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:11:32.700603 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:32.755141 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:32.848965 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:32.930557 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.059706 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.092753 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.128882 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.209635 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.237937 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.249182 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:33.251009 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:11:33.282493 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.383165 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.481024 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.599902 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.655572 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.723580 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.724910 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.727967 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.783581 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:33.806360 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.853890 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:33.856534 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:33.944327 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 14:11:33.984313 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(7693e3e204634676a758726bb6d3d2ce): perf score=0.033339
I20240501 14:11:34.019335 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(7693e3e204634676a758726bb6d3d2ce) complete. Timing: real 0.035s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":196,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1015,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":309,"threads_started":3,"wal-append.queue_time_us":195}
I20240501 14:11:34.022142 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(8147fae0de98429ab6554c7e7f5bf8ee): perf score=0.033338
W20240501 14:11:34.031023 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:11:34.031716 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:34.049551 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(8147fae0de98429ab6554c7e7f5bf8ee) complete. Timing: real 0.027s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":2010,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":1,"wal-append.queue_time_us":3054}
I20240501 14:11:34.051883 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(7693e3e204634676a758726bb6d3d2ce): 4921 bytes on disk
I20240501 14:11:34.052563 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(7693e3e204634676a758726bb6d3d2ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.054558 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(8147fae0de98429ab6554c7e7f5bf8ee): 4921 bytes on disk
I20240501 14:11:34.054924 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(8147fae0de98429ab6554c7e7f5bf8ee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.056779 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(66436017fd044f0b919629933ea56e9c): perf score=0.033400
I20240501 14:11:34.057509 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(49f2b0e67d3c43d584ac7583b98caeb9): perf score=0.033334
I20240501 14:11:34.115029 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(49f2b0e67d3c43d584ac7583b98caeb9) complete. Timing: real 0.057s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1662,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":505,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":23416}
I20240501 14:11:34.117306 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(49f2b0e67d3c43d584ac7583b98caeb9): 4581 bytes on disk
I20240501 14:11:34.118407 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(49f2b0e67d3c43d584ac7583b98caeb9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.137951 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(66436017fd044f0b919629933ea56e9c) complete. Timing: real 0.081s user 0.034s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":155,"dirs.queue_time_us":508,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":219,"threads_started":3,"wal-append.queue_time_us":13719}
I20240501 14:11:34.142434 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(66436017fd044f0b919629933ea56e9c): 4834 bytes on disk
I20240501 14:11:34.144469 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(66436017fd044f0b919629933ea56e9c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":635,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.146739 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(d178c3291efc44369e3adb921ccc6517): perf score=0.033403
I20240501 14:11:34.192968 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(d178c3291efc44369e3adb921ccc6517) complete. Timing: real 0.044s user 0.021s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":981,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":142,"threads_started":1,"wal-append.queue_time_us":255}
I20240501 14:11:34.196914 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(abac52c109174021ac11360b0d22ccfa): perf score=0.033400
I20240501 14:11:34.226548 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(abac52c109174021ac11360b0d22ccfa) complete. Timing: real 0.029s user 0.021s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":1684}
I20240501 14:11:34.228891 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(63a76e89d58d48ba9d1b3b98ef6c172d): perf score=0.033376
I20240501 14:11:34.256744 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(63a76e89d58d48ba9d1b3b98ef6c172d) complete. Timing: real 0.028s user 0.014s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1518,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":610,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":1,"wal-append.queue_time_us":265}
I20240501 14:11:34.258931 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(abac52c109174021ac11360b0d22ccfa): 4922 bytes on disk
I20240501 14:11:34.260280 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(abac52c109174021ac11360b0d22ccfa) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.262902 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(63a76e89d58d48ba9d1b3b98ef6c172d): 4581 bytes on disk
I20240501 14:11:34.263485 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(63a76e89d58d48ba9d1b3b98ef6c172d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.265548 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(d178c3291efc44369e3adb921ccc6517): 4581 bytes on disk
I20240501 14:11:34.266045 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(d178c3291efc44369e3adb921ccc6517) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240501 14:11:34.274132 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:34.388494 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:34.391852 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:34.519400 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:11:34.520210 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(25b5c40793b348bbb0e5251c8d553e30): perf score=0.033339
I20240501 14:11:34.539738 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(25b5c40793b348bbb0e5251c8d553e30) complete. Timing: real 0.019s user 0.014s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":999,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1466,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1358,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":170}
I20240501 14:11:34.541760 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(25b5c40793b348bbb0e5251c8d553e30): 4834 bytes on disk
I20240501 14:11:34.542153 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(25b5c40793b348bbb0e5251c8d553e30) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 14:11:34.565526 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:34.584029 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:34.605068 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:34.654431 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240501 14:11:34.658905 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(d64234effb3446bba1ff82bdf37acaa7): perf score=0.033394
W20240501 14:11:34.663925 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:34.689174 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(d64234effb3446bba1ff82bdf37acaa7) complete. Timing: real 0.030s user 0.024s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":166,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1025,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":427,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":295,"threads_started":3,"wal-append.queue_time_us":188}
I20240501 14:11:34.691884 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(d64234effb3446bba1ff82bdf37acaa7): 4581 bytes on disk
I20240501 14:11:34.692399 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(d64234effb3446bba1ff82bdf37acaa7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 14:11:34.694305 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(bd5489eab5b04751bc8b3e5fb9e8cf73): perf score=0.033385
I20240501 14:11:34.717965 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(bd5489eab5b04751bc8b3e5fb9e8cf73) complete. Timing: real 0.024s user 0.020s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":812,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":528}
I20240501 14:11:34.720443 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(bd5489eab5b04751bc8b3e5fb9e8cf73): 4832 bytes on disk
I20240501 14:11:34.721524 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(bd5489eab5b04751bc8b3e5fb9e8cf73) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":475,"lbm_reads_lt_1ms":4}
W20240501 14:11:34.742691 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 14:11:34.743343 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:11:34.745585 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:35.017848 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:35.190022 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:11:35.210716 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:35.391088 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:35.395992 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:35.430616 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:35.548205 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:35.565310 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:35.570636 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:35.632647 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:35.649386 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:35.721891 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:35.784641 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:35.821607 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:36.148891 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:36.155303 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:36.274258 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:36.274360 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:36.294945 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:36.346225 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:36.378407 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:36.404278 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:36.613277 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:36.623418 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:36.687793 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:36.891476 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(d64234effb3446bba1ff82bdf37acaa7): perf score=0.033339
W20240501 14:11:36.894922 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:36.904809 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(d64234effb3446bba1ff82bdf37acaa7) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1756,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":199,"threads_started":2,"wal-append.queue_time_us":152}
I20240501 14:11:36.907029 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(d64234effb3446bba1ff82bdf37acaa7): 4581 bytes on disk
I20240501 14:11:36.907459 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(d64234effb3446bba1ff82bdf37acaa7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:11:36.908953 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(56197e787ff14e85a07060f23e612498): perf score=0.033341
I20240501 14:11:36.922477 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(56197e787ff14e85a07060f23e612498) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1563,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":418,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":202}
I20240501 14:11:36.924391 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(56197e787ff14e85a07060f23e612498): 4834 bytes on disk
I20240501 14:11:36.924801 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(56197e787ff14e85a07060f23e612498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 14:11:36.927332 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:36.989925 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(0c3018b4356d462a9f2dd5511392661c): perf score=0.033376
I20240501 14:11:37.003706 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(0c3018b4356d462a9f2dd5511392661c) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":567,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1409,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":153}
I20240501 14:11:37.005514 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(0c3018b4356d462a9f2dd5511392661c): 4581 bytes on disk
I20240501 14:11:37.005880 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(0c3018b4356d462a9f2dd5511392661c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240501 14:11:37.061161 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:11:37.062659 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:37.150508 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:37.162245 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:11:37.177793 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(d01550698e8449b081a339f6ba30591c): perf score=0.033376
I20240501 14:11:37.189864 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(d01550698e8449b081a339f6ba30591c) complete. Timing: real 0.012s user 0.009s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1562,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":451,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":137}
I20240501 14:11:37.192121 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(d01550698e8449b081a339f6ba30591c): 4834 bytes on disk
I20240501 14:11:37.192553 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(d01550698e8449b081a339f6ba30591c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240501 14:11:37.404624 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:37.571370 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:37.767122 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:37.831140 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:37.858317 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:37.916164 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:37.969874 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:38.015498 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(1f1c8ff4149c42aa9517af442c38060c): perf score=0.033376
I20240501 14:11:38.029172 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(1f1c8ff4149c42aa9517af442c38060c) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":146}
I20240501 14:11:38.031082 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(1f1c8ff4149c42aa9517af442c38060c): 4921 bytes on disk
I20240501 14:11:38.031504 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(1f1c8ff4149c42aa9517af442c38060c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.149022 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:38.201136 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(40b29fb993f34e99ad3747a2ec7d6c4f): perf score=0.033389
I20240501 14:11:38.214372 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(40b29fb993f34e99ad3747a2ec7d6c4f) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1290,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":155}
I20240501 14:11:38.216157 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(40b29fb993f34e99ad3747a2ec7d6c4f): 4833 bytes on disk
I20240501 14:11:38.216519 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(40b29fb993f34e99ad3747a2ec7d6c4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.220069 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:38.251955 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:38.282598 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:38.335961 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:38.428045 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:38.432721 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:38.461230 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:38.484246 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:38.536701 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(d178c3291efc44369e3adb921ccc6517): perf score=0.033353
I20240501 14:11:38.549320 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(d178c3291efc44369e3adb921ccc6517) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":485,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":153}
W20240501 14:11:38.551362 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:38.551535 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(d178c3291efc44369e3adb921ccc6517): 4581 bytes on disk
I20240501 14:11:38.551935 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(d178c3291efc44369e3adb921ccc6517) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:11:38.569128 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(16324878552c4b6dbf10e8d5f73a0436): perf score=0.033343
W20240501 14:11:38.571231 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:38.583429 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(16324878552c4b6dbf10e8d5f73a0436) complete. Timing: real 0.014s user 0.008s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1298,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":171,"threads_started":2,"wal-append.queue_time_us":162}
I20240501 14:11:38.585022 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(16324878552c4b6dbf10e8d5f73a0436): 4834 bytes on disk
I20240501 14:11:38.585531 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(16324878552c4b6dbf10e8d5f73a0436) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.707597 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:38.721356 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(444c1a9d0e6943dc880161ae320673b6): perf score=0.033376
W20240501 14:11:38.723692 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:38.734825 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(444c1a9d0e6943dc880161ae320673b6) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2506,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":164,"threads_started":2,"wal-append.queue_time_us":147}
I20240501 14:11:38.736886 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(444c1a9d0e6943dc880161ae320673b6): 4921 bytes on disk
I20240501 14:11:38.737280 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(444c1a9d0e6943dc880161ae320673b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.739444 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:38.740097 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:38.805974 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(5648b7e0a1e744cea470feb7911d3d2c): perf score=0.033390
I20240501 14:11:38.818173 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(5648b7e0a1e744cea470feb7911d3d2c) complete. Timing: real 0.012s user 0.007s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1580,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":424,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":122,"threads_started":1,"wal-append.queue_time_us":202}
I20240501 14:11:38.820219 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(5648b7e0a1e744cea470feb7911d3d2c): 4581 bytes on disk
I20240501 14:11:38.820605 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(5648b7e0a1e744cea470feb7911d3d2c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240501 14:11:38.828104 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:38.898252 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:39.017045 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:39.041865 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:39.090587 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(68b04992f1f94b36b01dfde24a42318d): perf score=0.033364
W20240501 14:11:39.092613 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:11:39.113396 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(68b04992f1f94b36b01dfde24a42318d) complete. Timing: real 0.023s user 0.021s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1425,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":189,"threads_started":2,"wal-append.queue_time_us":164}
I20240501 14:11:39.115414 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(68b04992f1f94b36b01dfde24a42318d): 4581 bytes on disk
I20240501 14:11:39.115890 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(68b04992f1f94b36b01dfde24a42318d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.117736 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(eb336cffd30b447b880af97da6882d27): perf score=0.033350
I20240501 14:11:39.140383 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(eb336cffd30b447b880af97da6882d27) complete. Timing: real 0.022s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1918,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1923,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":133,"threads_started":1,"wal-append.queue_time_us":200}
I20240501 14:11:39.144066 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(eb336cffd30b447b880af97da6882d27): 4581 bytes on disk
W20240501 14:11:39.144086 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240501 14:11:39.144563 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(eb336cffd30b447b880af97da6882d27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.325925 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(7693e3e204634676a758726bb6d3d2ce): perf score=0.033396
I20240501 14:11:39.339232 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(7693e3e204634676a758726bb6d3d2ce) complete. Timing: real 0.013s user 0.004s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":127}
I20240501 14:11:39.341337 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(7693e3e204634676a758726bb6d3d2ce): 4921 bytes on disk
I20240501 14:11:39.342023 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(7693e3e204634676a758726bb6d3d2ce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":208,"lbm_reads_lt_1ms":4}
I20240501 14:11:39.343591 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(6628290de33e4aba8ccd1651d1d054dd): perf score=0.033334
W20240501 14:11:39.352561 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:11:39.355654 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(6628290de33e4aba8ccd1651d1d054dd) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1623,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":182144,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":163}
I20240501 14:11:39.357415 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(6628290de33e4aba8ccd1651d1d054dd): 4580 bytes on disk
I20240501 14:11:39.357739 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(6628290de33e4aba8ccd1651d1d054dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 14:11:39.386935 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:39.388238 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:39.569398 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:39.589604 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:11:39.649782 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling FlushMRSOp(01d9bd5e798f4bf2a5dd1eedcb4676cc): perf score=0.033335
I20240501 14:11:39.665916 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: FlushMRSOp(01d9bd5e798f4bf2a5dd1eedcb4676cc) complete. Timing: real 0.016s user 0.012s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1508,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2035,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":146}
I20240501 14:11:39.667716 15005 maintenance_manager.cc:419] P 8a13c028ba8a4a659b74af6c265df95e: Scheduling UndoDeltaBlockGCOp(01d9bd5e798f4bf2a5dd1eedcb4676cc): 4921 bytes on disk
I20240501 14:11:39.668056 14895 maintenance_manager.cc:643] P 8a13c028ba8a4a659b74af6c265df95e: UndoDeltaBlockGCOp(01d9bd5e798f4bf2a5dd1eedcb4676cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240501 14:11:39.692785 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:39.748999 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:39.754817 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:39.769127 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:39.797847 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:39.798717 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:11:39.985455 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:40.185591 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:40.234040 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:40.237330 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:40.272704 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:40.367770 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:40.454443 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:40.592242 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:40.597167 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:11:40.618450 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling FlushMRSOp(63a76e89d58d48ba9d1b3b98ef6c172d): perf score=0.033402
I20240501 14:11:40.630426 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: FlushMRSOp(63a76e89d58d48ba9d1b3b98ef6c172d) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1536,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":475,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":129}
I20240501 14:11:40.632032 15400 maintenance_manager.cc:419] P f23f426391724d6688aafffe7ee017d8: Scheduling UndoDeltaBlockGCOp(63a76e89d58d48ba9d1b3b98ef6c172d): 4581 bytes on disk
I20240501 14:11:40.632452 15290 maintenance_manager.cc:643] P f23f426391724d6688aafffe7ee017d8: UndoDeltaBlockGCOp(63a76e89d58d48ba9d1b3b98ef6c172d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 14:11:40.777580 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:40.974368 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:40.982872 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:40.994160 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.028992 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.067354 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.106643 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.174623 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.210060 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:41.214334 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.244450 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:41.264892 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:41.319896 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:41.393090 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:41.417737 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:41.727480 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:41.750325 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:11:41.759097 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling FlushMRSOp(7c778ae9911740c2b0cb42dfbc694bb1): perf score=0.033343
I20240501 14:11:41.771917 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: FlushMRSOp(7c778ae9911740c2b0cb42dfbc694bb1) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1819,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":130}
I20240501 14:11:41.773667 15269 maintenance_manager.cc:419] P 66e2dc34330848c3b8c2dd15898d34c8: Scheduling UndoDeltaBlockGCOp(7c778ae9911740c2b0cb42dfbc694bb1): 4834 bytes on disk
I20240501 14:11:41.773950 15158 maintenance_manager.cc:643] P 66e2dc34330848c3b8c2dd15898d34c8: UndoDeltaBlockGCOp(7c778ae9911740c2b0cb42dfbc694bb1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240501 14:11:41.879776 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:41.930802 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:42.075331 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:42.102531 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:11:42.281527 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:42.288383 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:42.338083 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:42.534785 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:42.538877 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:42.681567 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:42.695508 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:42.740350 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:42.890424 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:42.948626 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.164975 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:43.194582 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.255534 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.281692 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:43.415208 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:43.423174 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:11:43.478298 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.514813 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.530230 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.544950 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:43.560515 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.662285 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.668347 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:43.671285 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:43.680655 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:43.872735 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:43.909045 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:44.116418 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:44.135969 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:11:44.136705 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:44.272114 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:44.301049 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:44.335740 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:44.491328 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:44.494179 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:44.711488 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:44.722905 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:11:44.740649 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:11:44.795018 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:44.972839 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:45.020051 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:45.052527 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:11:45.078624 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:45.101972 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.124657 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:45.317663 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.455498 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:45.569779 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:45.580535 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.583092 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.692914 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.742806 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:45.743660 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.859972 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:45.953183 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:45.976641 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:46.010358 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:46.063625 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:46.073989 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.099694 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.114817 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:46.131518 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:46.248720 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:46.373073 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.501122 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.511070 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.561820 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:46.774016 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.998809 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:46.999735 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:47.068687 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:47.073343 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:11:47.113806 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:47.241256 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:47.503695 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:47.662712 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:47.678418 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:47.731096 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:47.787550 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:47.883888 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.054837 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 14:11:48.055657 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:48.115690 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:48.116194 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:48.173053 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.305861 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:48.307466 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.385860 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.390451 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.437599 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.443305 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:11:48.491091 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:48.501644 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.540159 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:48.569976 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.606817 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.663565 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:48.774766 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:48.944242 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:48.963740 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:48.994575 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:49.232767 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:49.238759 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:49.340751 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:11:49.362679 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:49.514578 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:49.534194 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:49.541360 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:49.574821 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:49.689646 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:49.814610 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:49.879611 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:49.910502 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:11:50.042659 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:50.069537 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:50.088160 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:50.126652 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:50.155946 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:50.227768 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:50.286213 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:50.456316 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:11:50.485172 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:50.500779 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:50.646132 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:50.656038 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:50.812027 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:50.822233 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:50.872956 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:51.011060 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.027117 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:51.043510 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.072800 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.107967 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.121107 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.139189 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:51.230705 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:51.251803 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.436599 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.445765 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:51.467242 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:51.700834 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:51.715570 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:51.762751 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:52.167073 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:52.196547 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:52.242427 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:52.322285 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:11:52.361171 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:52.521466 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:52.527072 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:52.685037 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:52.719609 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:52.800750 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:52.832865 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:52.943668 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.104617 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.144008 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:53.222160 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:53.329780 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:53.343722 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:53.462260 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.483049 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:11:53.483683 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.524752 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.529186 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:53.534960 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:53.578006 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.651183 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.663342 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:53.677678 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:53.902837 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:53.953805 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:54.058395 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.082239 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.117923 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:54.195822 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:54.440579 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:11:54.443711 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:54.486564 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.539186 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:54.545454 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.640336 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.750401 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:11:54.873498 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.942510 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:54.949891 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:54.965035 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:55.091058 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:55.120958 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:11:55.121573 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.209285 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:11:55.253070 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:55.373077 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.544889 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:11:55.545467 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:55.659502 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.674281 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:55.811436 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.826766 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:55.857666 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.884840 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.916575 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:55.929775 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:55.952527 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:56.134338 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:56.251294 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:56.281948 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:56.319137 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:56.487619 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:56.493019 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:56.518028 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:56.535370 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:56.671586 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:56.690672 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:56.831737 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:57.102169 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:57.110808 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:57.197818 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:57.245060 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:57.429433 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:57.463667 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:57.491569 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:11:57.552273 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:57.711357 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:57.739434 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.040369 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.108528 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:58.128115 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.228549 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.238893 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:58.242300 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.296371 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.339615 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.383390 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:58.455284 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:58.483172 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:58.494172 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:11:58.535382 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:58.586694 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:58.637095 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:58.702611 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:58.824674 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:59.012990 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:59.139153 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:59.143905 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:59.222046 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:59.473320 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:11:59.512966 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:11:59.513597 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:59.516270 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:59.676240 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:59.723831 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:11:59.797638 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:11:59.848675 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:59.928066 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:11:59.982165 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:00.053463 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:00.073772 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.206789 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:00.210539 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:00.217398 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:00.231706 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:00.232308 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.438419 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.506327 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.569861 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.657243 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:12:00.757757 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.805140 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:00.805434 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:00.832803 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:00.947062 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:01.021236 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.064154 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.134647 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.280773 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.331831 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.341506 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:01.402974 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:01.424974 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:01.572328 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.627861 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:01.649569 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:01.942881 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:01.945436 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.241303 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:02.249648 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.265086 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.310760 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:02.418124 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:02.486991 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:02.641047 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:02.663383 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.751847 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.804724 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.851356 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:02.921546 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:03.029019 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:03.069703 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:03.315783 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:03.327394 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:03.469075 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:03.574482 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:03.577816 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:03.578388 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:03.611130 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:03.646306 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:03.701835 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:03.711333 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:03.877734 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:03.901298 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:04.033658 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:04.039816 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:04.060426 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:04.189496 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:04.272889 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:04.438410 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:04.513140 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:04.527362 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:12:04.578147 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:04.584830 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:04.694818 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:04.891130 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:04.933224 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:04.938041 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.029408 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.086843 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:12:05.088531 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.171810 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.246599 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:05.260957 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:05.312558 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.316901 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:05.372036 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.386924 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:05.399521 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.465411 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.518491 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:05.595657 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:05.823850 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:05.829665 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:05.871071 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:05.881534 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:05.895401 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:05.944577 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:06.265343 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:06.270332 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:06.320950 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:06.352847 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:06.409092 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:06.410176 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:06.504547 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:06.619287 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:06.823725 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:06.827399 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:06.971022 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:07.026654 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.310240 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:07.358630 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:07.468194 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.568550 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:07.670578 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.712707 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:12:07.806111 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.858539 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.881377 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.895934 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.916936 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:07.952306 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:08.002185 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:08.008966 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:08.069128 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:08.230947 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:08.373003 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:08.407506 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:08.418849 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:08.445425 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:08.531019 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:08.589632 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:08.806973 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:08.850036 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:08.850767 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:08.858566 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:08.919791 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:08.989590 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:09.024818 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:09.056262 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:09.392105 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:09.490203 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:09.533497 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:09.601598 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:09.624449 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:09.711330 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:09.753732 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:09.947600 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:09.974370 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:10.091289 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.270924 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:10.304517 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.313458 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:12:10.315922 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.357612 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.375757 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:10.427858 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:10.551661 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:10.552269 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.611645 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.640501 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.666965 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:10.692080 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:10.692696 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:10.806380 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:10.808151 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:10.828459 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:10.868218 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:10.935777 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:11.207268 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:11.219645 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:11.280779 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:11.327201 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:11.380285 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:11.466317 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:11.526722 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:11.653913 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:11.765304 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:11.826100 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:11.923271 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:12.092576 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:12.105001 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:12.119225 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:12.300693 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:12.588582 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:12:12.620729 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:12.621389 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:12.680292 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:12.801980 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:12.804252 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:12.992974 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:13.047338 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.134166 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.169718 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.198597 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.234323 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:13.241683 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.256726 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:13.259984 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.261445 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:13.482957 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:13.653581 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:12:13.675500 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:13.807036 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.809655 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:13.815390 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:13.830363 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:13.999472 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:14.006841 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:14.166568 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:14.263842 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:14.325968 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:14.344578 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:14.482195 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:14.505704 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:14.520570 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:14.606484 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:12:14.752619 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:14.852416 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:15.039839 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.044884 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.187938 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:15.401372 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:15.444288 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:15.468636 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:12:15.469139 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.489563 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:15.580576 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:15.624176 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.738416 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.794251 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.802343 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:15.812899 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:15.838013 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.878998 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:12:15.879793 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:15.917486 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 14:12:15.918012 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:15.957513 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:15.985328 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:15.990315 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:16.109561 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:16.283736 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:16.398658 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:16.409094 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:16.581419 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:16.590013 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:16.688861 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:16.771334 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:16.839841 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:16.848685 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:16.990761 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:17.012073 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:17.123044 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:17.391341 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:17.526824 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:17.599028 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:17.673386 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:17.860201 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:18.005556 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:12:18.082106 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.130892 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.137557 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.160741 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.181829 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.200898 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.202857 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.227368 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.280309 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:18.440276 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.532259 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.623929 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.702608 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:18.729239 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.754657 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:18.791834 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.809350 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:18.850921 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:18.924515 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:19.110988 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:19.166219 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:19.184829 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:19.393766 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:19.425685 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:19.457978 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:19.487303 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:19.495313 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:19.531261 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:19.624819 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:19.628732 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:19.917732 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:20.044394 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:20.051913 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:20.222285 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:20.482352 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:12:20.527878 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:20.532367 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:20.590181 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:20.593955 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:20.606432 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:20.656992 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:20.685029 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:20.685882 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:20.779680 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:20.822885 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:20.930356 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:20.931079 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:20.963955 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:20.993453 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.034402 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:21.072319 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.103258 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:21.201012 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:21.202441 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:21.209792 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:21.406272 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.484625 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.556499 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:21.603148 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.705163 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:21.773589 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.894909 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:21.900118 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:22.058568 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:22.087787 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:22.241524 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:22.295177 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:22.435196 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:22.572625 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:22.682230 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:22.812186 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:22.904735 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:22.979810 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:12:23.081254 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:23.105101 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.130775 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:23.183616 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.303212 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.304926 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.374878 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:23.412623 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:23.609598 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.692761 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:23.695705 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:23.710664 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.748131 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:23.769680 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:23.805356 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:23.813622 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:23.852869 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:24.054842 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:24.073052 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:24.094695 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:24.283648 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:24.300864 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:24.438979 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:24.537950 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:24.566126 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:24.606178 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:24.754240 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:24.860507 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:12:24.861157 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:24.897217 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:24.961911 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:12:25.155094 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:25.412104 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:25.428908 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:25.449033 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:12:25.511864 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:25.568315 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:25.597738 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:25.662705 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:25.706337 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:25.819104 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:25.830291 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:25.840302 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:25.927079 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:25.958230 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:25.991974 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.129657 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.174939 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:26.185598 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:26.239329 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:26.251963 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:26.335793 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.509299 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.510411 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:26.521162 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:26.536490 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:26.550143 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.631131 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:26.697742 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.721100 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:26.725992 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:26.771828 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:27.042961 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:27.166280 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:27.359436 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:27.401612 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:27.478173 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:27.555166 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:27.806391 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:27.967546 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:27.977885 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:27.979547 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:28.016685 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.083963 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.147120 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:12:28.327062 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.444442 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:28.473587 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:28.523954 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.528023 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.668962 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.724890 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:28.725370 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.746415 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:28.846771 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:28.883693 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:28.893010 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:28.898828 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:28.999199 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:29.072398 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:29.170354 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:29.226465 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:29.242940 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:29.250072 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:29.390722 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:29.539824 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:29.547838 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:29.593963 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:29.746389 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:29.776083 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:29.798238 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:30.058432 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:12:30.082612 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:12:30.282265 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:30.315682 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:30.477728 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:30.503701 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:30.535604 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:30.575984 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:30.619693 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:30.758831 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:30.803369 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:12:30.848816 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:30.875613 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.020383 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:31.021121 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.075587 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:31.076216 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.081504 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.170871 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.203598 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:12:31.204363 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:31.261477 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.265311 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.277535 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.384006 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.505538 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.531450 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.605947 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.637995 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:31.819990 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:31.849166 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:31.953655 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:32.047333 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:32.098529 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:32.179895 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:32.210587 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:32.436326 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:32.492583 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:32.733623 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:32.889564 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:32.942817 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:32.946542 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:32.985754 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.094673 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.243237 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:33.277824 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:33.377900 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:12:33.383944 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.493254 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.524083 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.537817 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.540946 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:33.587131 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.692641 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:33.733270 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.776492 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:33.780460 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:33.856431 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:33.966459 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:34.073195 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:34.093451 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:34.114719 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:34.193019 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:34.247017 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:34.437693 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:34.576238 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:34.639703 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:34.767207 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:34.816002 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:34.877786 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:34.928505 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:35.095387 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:35.115149 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:35.210597 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:12:35.311692 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:35.316160 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:35.382505 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:35.638926 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:35.651451 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:35.668536 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:12:35.700984 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:35.711189 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:35.812441 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:35.936077 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:35.965703 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:35.967967 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:36.001017 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:36.072496 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:36.106431 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:36.237411 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:36.261328 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:36.283113 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:36.306948 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:36.344779 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:36.353796 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:36.402175 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:36.407804 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:36.408425 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:36.455631 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:36.506601 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:36.709184 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:36.719820 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:36.844826 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:37.075490 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:37.158777 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:37.252765 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:37.403442 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:37.492024 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:37.558486 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:37.607582 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:37.625221 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:37.906606 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.041311 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:38.076857 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:38.094656 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.163522 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.251184 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:12:38.302919 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:38.373106 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:38.469555 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.514768 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:38.569967 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:38.613276 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.625146 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.654016 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.724642 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.743649 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:38.859629 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:38.898871 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:38.949036 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:38.968283 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:39.018594 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:39.123302 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:39.181370 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:39.402985 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:39.425125 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:39.430002 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:39.488878 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:39.648677 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:39.753944 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:39.756305 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:39.967672 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:40.115494 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:40.119313 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:40.204443 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:40.216856 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:12:40.227943 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:40.351711 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:40.508458 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:40.527479 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:12:40.585793 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:40.684129 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:40.812770 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:40.828136 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:40.875617 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:40.981257 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:40.998220 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.026647 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.134411 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:41.143718 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.223868 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:41.257000 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.262094 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.271816 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:41.272601 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:41.320546 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:41.411471 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:41.422627 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:41.423420 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:41.449457 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:41.545964 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.659130 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:41.691617 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:41.787765 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:41.919385 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:42.068090 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:42.121102 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:42.209501 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:42.247808 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:42.263063 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:42.613847 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:42.693488 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:42.695474 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:42.810230 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.062526 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.085748 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:43.088968 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.207741 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:12:43.255501 12759 debug-util.cc:398] Leaking SignalData structure 0x563c6d7cd0e0 after lost signal to thread 29760
W20240501 14:12:43.314440 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:43.327266 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.435281 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:43.449995 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.534624 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.602669 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:43.716620 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.719153 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:43.744815 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:43.817387 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.819576 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:43.956001 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:44.005980 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:44.025766 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:44.034499 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:12:44.054080 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:44.060916 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:44.238128 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:44.314358 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:44.327708 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:44.392784 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:44.410063 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:44.449208 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:44.747061 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:44.858955 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:45.132321 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:45.173424 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:45.174185 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.335198 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:45.339238 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:45.375269 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.475150 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:45.567054 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.568490 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:45.609781 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.718792 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:45.729931 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.838675 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:12:45.872761 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.930230 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:45.943400 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:45.961357 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:46.027122 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:46.089155 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.126005 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:46.328215 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 14:12:46.328939 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.381783 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.387547 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.413383 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:12:46.444746 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:12:46.479655 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:46.499783 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:46.523138 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:46.555867 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:46.624866 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:46.663218 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.785796 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:46.832206 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.896059 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:46.936856 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:47.184218 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:47.423934 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:47.452939 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:47.468416 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:47.576409 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:47.847870 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:47.892485 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:47.894810 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:48.070547 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 14:12:48.071373 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:48.198009 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.233812 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.264143 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:12:48.270504 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.330045 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:48.333441 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.343186 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.378572 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:48.562386 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.609256 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:48.739032 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:48.781479 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:48.835253 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:48.862471 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:48.863534 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:49.040163 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:49.056795 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:49.060551 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:12:49.063154 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:49.126156 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:49.158951 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:49.228005 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:49.248359 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:49.297817 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:49.314401 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:49.560834 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:49.610250 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:49.646646 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:49.827920 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:50.128031 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:50.206390 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:12:50.227094 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:50.275444 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:50.276755 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:50.405629 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:50.471470 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:50.473870 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:12:50.594314 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:50.659822 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:50.697126 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:50.775337 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:50.787561 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:50.801739 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:12:50.961750 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.006438 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:51.104195 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.189397 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:51.320372 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:51.332008 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:51.378188 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:51.401648 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.406435 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:51.449564 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:51.455925 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:51.456509 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.533047 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:51.554222 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:12:51.554919 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.628970 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:12:51.629595 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.706686 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.761202 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:51.798754 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:51.854905 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:52.145081 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:52.307847 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:52.507308 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:52.507423 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:52.693957 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:52.755530 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:52.779188 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:52.860977 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:52.914784 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:52.966990 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:53.078768 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:12:53.168344 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.233278 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.296093 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:53.343330 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.514928 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.557709 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:53.671663 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:53.697878 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.792451 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:53.867718 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.893471 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:53.904399 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.916494 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:53.943593 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:54.122162 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:54.150127 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:54.177896 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:54.286576 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:54.322407 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:54.324229 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:54.333900 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:54.375404 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:54.749352 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:54.851766 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:54.885905 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:54.906402 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:55.000491 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:55.186642 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:55.193285 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:55.261369 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:12:55.262217 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:55.361675 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:55.413766 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:55.423626 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:55.556571 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:12:55.557040 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:55.789076 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:55.917011 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:55.938819 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:56.006932 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.065869 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:56.088976 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.129910 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:56.172588 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:56.307142 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:56.395094 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:56.409849 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:56.432951 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.476629 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:56.487514 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:12:56.488185 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.499585 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.511843 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:56.625481 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:12:56.626138 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.632019 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.816103 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:56.821817 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:12:56.822455 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:57.226840 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:57.230646 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:57.340814 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:57.418308 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:57.527309 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:57.573057 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:57.666975 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:57.682441 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:57.684854 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:57.837672 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:57.928687 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:12:58.013094 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:58.060781 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:58.071504 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:58.246379 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:58.427160 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:58.536685 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:58.639541 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:58.692806 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:58.705119 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:12:58.717240 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:58.747640 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:58.750053 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:58.878791 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:58.968112 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:59.016572 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:59.037310 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:59.100163 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:59.134001 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:12:59.146968 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:59.166451 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:59.250586 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:59.277273 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:12:59.544360 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:59.557938 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:59.740712 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:12:59.945708 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:12:59.968885 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:00.028507 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:00.045348 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:00.095002 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:00.190169 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:00.223464 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:00.295538 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:00.313974 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:00.410990 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:00.480134 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:00.558059 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:13:00.668020 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:00.705107 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:00.835016 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:00.921666 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:01.034199 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.060873 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.062467 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:01.176915 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.200261 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:01.232966 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:01.247004 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:01.411576 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.440773 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:01.552953 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.626361 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:01.640667 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:13:01.650408 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.765499 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:01.788066 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:01.873463 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:01.894424 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:01.912766 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:01.955108 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:02.044476 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:02.143430 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:02.166918 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:02.401481 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:02.410213 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:02.458388 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:02.558537 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:02.607970 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:02.675812 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:02.753943 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:02.783867 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:02.802410 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:02.895067 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:02.993678 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:03.155606 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:03.214872 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:03.304505 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:03.411190 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:03.416838 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:03.426456 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:03.634723 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:03.750427 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:03.813354 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:03.815407 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:03.886826 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:03.914312 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:04.135651 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:04.144685 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:04.145187 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:04.241217 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:04.243826 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:04.410387 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:04.483397 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:04.644596 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:04.652902 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:04.684762 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:04.720855 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:04.799472 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:05.025225 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:05.034857 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:05.068120 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:05.086203 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:05.097121 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:05.113652 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:05.178804 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:13:05.250795 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:05.452764 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:05.584319 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:05.686115 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:05.819514 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:13:05.857846 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:05.916005 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:05.975334 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:05.976929 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.038553 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.088342 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.161876 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.212023 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:06.286698 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:06.382117 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:06.440339 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.534618 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:06.541467 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.651296 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:06.657163 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:06.691581 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:06.699079 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:06.762861 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:06.842062 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:06.922842 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:07.034157 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:07.051306 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:07.066426 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:07.162367 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:07.180846 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:07.249538 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:07.382119 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:07.394299 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:07.463903 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:07.481375 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:07.563628 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:07.633610 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:07.699232 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:07.895128 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:13:08.140364 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:08.182876 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:08.253218 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:08.365016 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:08.365366 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:08.412361 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:08.608166 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:08.668175 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:08.678490 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:08.705684 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:08.787554 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:09.170022 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:13:09.170315 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:09.170627 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:09.295357 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:09.312772 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:09.316473 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:09.448251 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:09.451119 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:09.475368 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:09.569305 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:09.578063 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:09.585990 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:09.825891 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:09.835642 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:09.856978 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:09.869367 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:09.908304 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:09.938249 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:10.047245 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:10.121035 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:10.305583 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:13:10.491336 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:10.596172 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:13:10.596697 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:10.620452 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:10.706914 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:10.891878 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:10.917795 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:13:10.936378 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:10.995324 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:11.068529 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:11.233094 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:11.233918 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:11.294761 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:11.328382 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:11.372020 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:11.551062 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:11.569402 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:11.608628 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:11.760293 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:11.761039 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:11.765339 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:11.886682 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:11.896138 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:11.934316 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:11.935078 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:12.040944 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:12.121992 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:12.129606 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:12.144979 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:12.159924 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:12.190008 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:12.339347 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:12.355275 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:12.509033 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:12.512897 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:13:12.548605 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:12.560263 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:12.566816 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:12.900933 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:12.918913 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:13.245488 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:13.293179 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:13.332614 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:13.369390 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:13.487885 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:13.518029 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:13.560775 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:13.592830 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:13.770949 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:13.829984 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:14.112376 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:14.124543 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:14.180963 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:13:14.181617 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:14.222424 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:14.253731 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:14.313706 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:14.358317 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:14.612067 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:14.695323 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:14.705516 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:14.725860 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:14.748157 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:14.757315 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:14.815488 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:13:14.957115 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:14.973260 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:15.005525 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:15.097012 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:15.137816 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:15.274158 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:15.311345 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:15.313005 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:15.737332 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:15.770009 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:15.800123 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:15.820668 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:15.827899 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:15.837674 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:16.006701 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:13:16.007586 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:16.059603 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:16.172323 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:16.192685 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:16.440122 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:16.467207 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:16.537945 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:16.566164 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:16.594666 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:16.631796 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:16.767850 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:16.902913 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:16.929589 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:16.934538 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:16.987485 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:16.999706 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:17.029284 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:17.223776 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:17.224395 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:17.271780 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:17.334394 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:17.388720 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:13:17.455682 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:17.467584 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:17.473539 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:17.512943 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:17.586038 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:17.759378 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:17.793385 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:17.858036 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:18.056684 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:18.183822 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:18.258059 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:18.259544 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:18.373387 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:18.495903 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:18.661914 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:18.744622 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:18.766535 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:18.777319 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:18.954339 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.048157 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:19.085544 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.123541 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:19.189332 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:19.273869 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:19.537153 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:19.671473 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:19.698069 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:19.699604 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:19.706676 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.707178 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:19.786970 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:19.800257 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.868858 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.870242 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.899097 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:13:19.937371 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:19.948460 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:20.251013 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:20.455857 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:20.469089 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:20.573460 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:20.595629 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:20.601080 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:20.665350 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:20.713716 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:20.766216 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:20.911831 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:21.158593 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:21.162698 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:13:21.163306 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:21.217235 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:21.294756 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:21.296204 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:21.379144 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:21.481218 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:13:21.517761 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:21.517951 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:21.622668 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:21.769002 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:22.011986 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:22.025770 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:22.064486 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:22.116899 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:13:22.117621 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:22.178294 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:22.237896 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:22.267338 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:13:22.268188 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:22.278278 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:22.322939 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:22.404331 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:22.518510 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:22.549453 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:22.596211 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:22.684971 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:13:22.728237 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:23.007107 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.016191 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.052588 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:23.098205 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.232126 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.323776 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.331233 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:23.417616 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.633528 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:23.652669 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:23.686235 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:23.836199 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:23.861932 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:23.889015 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:23.955585 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:24.059577 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:24.065089 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:24.385543 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:13:24.442397 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:24.506032 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:24.513060 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:24.564616 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:24.660090 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:24.669132 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:24.688899 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:24.969719 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:24.998917 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.023834 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:25.038029 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.123987 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:13:25.230051 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:25.308023 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.357396 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.446532 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.585670 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:25.644891 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.675077 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:25.891614 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:25.929791 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:25.954612 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:26.048460 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:26.048574 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:26.055142 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:26.100804 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:26.330801 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:13:26.333339 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:26.344805 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:26.395181 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:26.494699 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240501 14:13:26.495384 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:26.593891 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:26.604254 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:26.652910 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:26.887974 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:26.982378 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:27.027357 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:27.033591 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:27.052361 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:27.052989 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:27.201836 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:27.375247 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:27.403069 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:27.508898 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:27.521920 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:27.539435 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:27.567926 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:27.567972 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:27.569839 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:27.584118 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:13:27.688681 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:27.935181 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.018903 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:28.054831 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.145347 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.351576 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:28.354604 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.463130 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.533869 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:28.619446 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:28.704977 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.754943 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.786900 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:28.857659 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.870817 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:28.877761 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:28.958673 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:29.051185 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:29.113328 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:29.333947 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:29.452014 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:13:29.520113 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:29.571285 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:29.694332 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:29.714076 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:29.752164 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:29.816490 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:29.893424 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:13:29.923172 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.037287 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:30.062821 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:30.075661 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.084698 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:30.091943 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.443758 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.566256 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.632187 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.689710 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:30.690390 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:30.874576 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:30.955755 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:31.122627 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:31.198522 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:31.249315 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:31.268896 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:31.273797 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:31.386101 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:13:31.410583 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:31.430483 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:31.440550 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:31.450937 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:31.536914 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:31.600225 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:31.600909 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:31.679358 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:31.876196 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:31.887607 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:31.892817 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:32.064560 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:32.155162 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:32.216837 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:32.242754 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:32.326174 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:13:32.338202 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:32.339125 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:32.458452 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:32.465242 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 14:13:32.588579 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:32.613955 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:32.662581 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:32.740070 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:32.821451 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:32.898100 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:32.947634 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:33.043774 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:33.118209 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:33.135318 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:33.319813 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:33.504508 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:33.587323 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:33.659415 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:33.808934 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:33.858978 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:33.877429 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:33.936671 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:34.035909 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:34.108069 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:34.158404 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:34.274843 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:34.322626 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:34.376736 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:34.414796 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:34.482959 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:34.551663 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:13:34.604025 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:34.689425 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:34.781661 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:13:34.821066 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:34.874147 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:34.882750 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:34.966691 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:35.022553 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.025013 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.422899 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:35.484339 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:35.508800 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:35.532218 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.620131 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.661244 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.807576 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.817214 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:35.914440 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:35.920483 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:36.028172 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:36.068219 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:36.277014 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:36.298493 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:36.402910 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:13:36.486258 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:36.573493 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:36.682404 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:36.683096 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:36.727952 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:36.748073 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:36.949131 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:37.018167 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:37.034636 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:37.088114 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:37.109181 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:37.159857 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:37.278942 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:37.293638 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:37.355525 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:37.448799 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:13:37.552639 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:37.555084 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:37.555701 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:37.564980 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:37.629002 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:37.649080 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:37.783071 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:37.933964 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:38.047297 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:38.056898 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:38.063506 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:38.128317 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:38.240550 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:38.285162 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:38.380336 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:38.401676 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:38.552891 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:38.599128 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:38.806246 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:38.865869 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:39.024457 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:39.062935 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:39.124256 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:39.191084 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:39.234496 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:39.345307 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:39.358842 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:39.458964 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:39.489848 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:39.566418 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:39.575454 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:39.597544 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:39.672379 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:39.925426 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:39.941677 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:39.972379 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:40.008734 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:40.052320 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.099673 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.105903 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:13:40.231909 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:40.247079 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.536073 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.540489 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:40.625439 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.764828 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.769428 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.776830 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:40.838815 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:40.964704 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:41.029803 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:41.099088 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:41.341825 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:41.402973 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:13:41.417261 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:41.546741 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:41.550750 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:41.655630 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:41.664631 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:41.745613 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:41.798295 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:41.827059 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:41.844868 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:41.921793 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:41.957109 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:41.957804 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:42.087560 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:42.178534 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:42.191977 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:42.273947 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:42.297829 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:42.334965 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:42.348635 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:42.374034 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:42.517874 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:42.567749 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:13:42.697743 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:42.740783 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:42.760509 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:42.805393 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:42.983294 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.101990 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:13:43.114343 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.137526 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.148586 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:43.280738 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:43.342954 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.417493 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.640591 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:43.855926 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:43.875484 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.940618 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:43.944010 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.131016 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:44.186287 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.206460 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:44.260140 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.432075 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:44.454807 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.507763 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:44.603675 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.629362 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.651422 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:13:44.733443 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:44.752600 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:44.752856 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.806334 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:44.831699 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:13:44.856263 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:44.950418 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:45.016489 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:45.089860 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:45.383857 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:45.448593 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:45.484098 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:45.568928 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:45.593798 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:45.629091 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:45.770413 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:45.873431 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:46.042521 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:46.112329 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:46.150051 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:46.380051 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:46.416802 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:13:46.419225 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:46.502264 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:46.511466 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:46.534356 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:46.592386 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:46.643378 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:46.681866 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:46.919768 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:47.019935 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:47.036134 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:47.106984 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:47.122365 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:47.128248 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:47.154314 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:47.249053 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:47.250315 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:47.311486 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:47.327488 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:47.390856 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:13:47.443650 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:47.567375 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:47.573015 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:47.711170 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:47.769198 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:47.977844 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:48.005497 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:48.030814 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:48.049582 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:48.065860 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:48.086561 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 14:13:48.087385 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4bd8d72845814f17961145fd8d4d0bbd: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:48.102913 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:48.235296 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:48.308053 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:48.720052 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:48.790817 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:48.833994 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:48.968600 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.050973 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.086609 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:49.205911 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:49.230655 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.310981 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:49.321178 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:49.326386 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:49.326961 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.359102 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:49.477283 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.550043 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.569444 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.606163 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d381cfd9ae54c03a044567453d2d91c: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:49.673803 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 14:13:49.721053 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:49.744931 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:49.774363 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:49.853206 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:49.894584 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:49.901386 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:49.959869 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:50.053766 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:13:50.326654 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:50.418268 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:50.468640 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:50.497359 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:50.685835 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:50.822422 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:50.886484 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:50.926416 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:51.144251 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:51.211570 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:51.218272 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:51.244762 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:51.437041 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:51.518656 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:51.532148 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:51.657711 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:51.746374 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:51.851164 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:51.875671 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:51.988024 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:52.012732 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:52.020726 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:52.031335 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:52.130921 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:52.175894 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:52.190452 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:52.207828 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:52.225999 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:52.229938 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:52.288590 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:52.348289 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:52.495754 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:52.531080 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:52.553386 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:52.591063 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:13:52.720023 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:13:52.819957 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:52.819957 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:52.959290 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:52.992395 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:53.054231 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:53.155702 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:53.219251 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:53.291200 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:13:53.352800 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:53.355388 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:53.392850 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:53.622318 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:53.887681 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:53.948952 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:53.963726 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:54.046051 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:54.077368 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:54.313602 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:54.394604 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:54.435001 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:54.480620 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:54.545876 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:54.555354 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:54.568118 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:54.576234 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:54.584213 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:54.641181 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:54.674232 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:54.758196 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:13:54.762311 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:54.869918 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:54.918965 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:55.015127 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:55.031652 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:55.049216 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:13:55.410822 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:55.416990 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:55.450502 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:55.493547 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:55.697100 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:55.736781 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:55.799269 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:55.917155 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:55.930243 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:56.082321 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:56.342948 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:56.356179 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:56.370430 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:56.453895 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:13:56.454489 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:56.536339 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:56.657809 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:56.886585 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:56.887053 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:56.928031 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.000196 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:57.033489 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.054828 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.115465 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:57.123610 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.214430 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.296852 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:57.301220 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.304793 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:57.392676 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:57.409242 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:57.444504 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:57.516013 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:57.530313 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:13:57.584353 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:13:57.774464 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:57.828894 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:57.883018 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:57.963783 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:57.965122 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:13:57.975983 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:58.204916 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:58.221453 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:58.321041 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:13:58.325666 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:58.388379 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:58.394099 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:58.564314 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:58.763340 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:58.925284 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:59.066872 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.148548 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:59.167950 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.211215 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:59.433521 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:13:59.471661 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.497896 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:59.514542 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.550769 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:13:59.558355 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.713910 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.739998 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:59.799470 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:13:59.826777 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.877890 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:59.907478 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:13:59.921280 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:13:59.948048 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:13:59.949041 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:13:59.980470 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:13:59.998986 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:00.293174 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:00.394786 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:00.512822 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:00.633780 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:00.747478 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:00.758504 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:00.791857 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:00.872794 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:01.027915 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:01.246930 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:01.393011 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:01.421629 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:01.499657 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:14:01.502949 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:01.512653 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:01.541622 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:01.604104 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:01.693151 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:01.809706 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:01.842598 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:01.917625 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:01.922080 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:01.991513 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:02.117275 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:02.232049 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:02.232993 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:02.278856 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:02.305771 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:02.343124 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:02.363502 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:02.460894 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:02.541687 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:02.598237 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:02.687644 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:14:02.688303 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:02.713353 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:02.724390 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:03.005098 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:03.065605 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:03.066211 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:03.105703 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:03.271510 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:03.363469 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:03.426198 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:03.428788 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:03.451599 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:03.639683 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:03.761380 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:03.799248 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:03.948642 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:04.026309 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:04.232604 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:04.278889 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:04.316128 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:04.341846 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:04.413167 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:04.508147 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:04.584132 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:04.593490 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:04.598498 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:04.658867 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:04.773717 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:04.872838 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:04.944628 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:04.945878 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:04.974339 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:14:05.036923 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:14:05.051937 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:05.072425 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:05.075366 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:05.079623 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:05.180929 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:05.192075 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:05.535049 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:05.680172 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:05.692607 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:05.737483 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:06.065517 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:06.107836 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:06.150710 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:06.184239 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:06.227348 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:06.294556 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:06.362120 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:06.509094 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:14:06.588471 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:14:06.752125 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:06.759104 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:06.796801 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:06.916633 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:07.016664 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:07.115206 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.116187 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.189872 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:07.206116 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.244428 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.247246 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:07.294186 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:07.326120 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:07.335125 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:07.444679 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.577618 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:14:07.591917 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:07.630219 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.658041 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:07.727867 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:07.852962 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:07.868422 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:08.152822 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:08.159744 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:08.160373 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:08.272390 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:08.332903 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:08.456899 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:14:08.457737 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:08.548168 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:08.571800 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:08.678261 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:08.690472 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:08.746261 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:08.834206 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:09.148397 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:09.256908 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.262483 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.273552 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:09.371522 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:09.438045 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:09.438361 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.553134 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:09.641378 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:09.653276 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.674100 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.729678 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.790431 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:09.834457 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:09.935604 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:09.939486 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:10.008611 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:14:10.054616 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:10.137653 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:14:10.138927 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:10.244573 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:10.252523 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:10.363509 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:10.662166 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:10.723933 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:10.755250 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:10.899237 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:10.915997 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.058050 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:11.114796 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:11.165927 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.189806 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.423445 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:11.558615 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:11.559458 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.611011 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:11.700238 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:11.728742 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.763063 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:11.766440 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.842099 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:11.925113 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:11.951257 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:12.003347 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:12.051659 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:12.226444 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:12.255677 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:12.387845 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:12.403352 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:12.458719 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:12.523995 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:14:12.545805 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:12.554205 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:12.644752 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:12.693343 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:12.735406 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:12.736142 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:12.755149 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.076748 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:13.106895 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.150252 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.214432 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.296990 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:13.308585 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:13.316180 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:13.647531 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:14:13.648195 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:13.751998 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.755717 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.815847 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:13.846333 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:13.956812 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:14.008322 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:14.098541 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:14.098834 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:14.183199 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:14.224254 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:14.246093 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:14.598476 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:14.629235 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:14.635032 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:14.651015 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:14.755719 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:14.788967 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:14.888406 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:14:14.988408 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:14.999845 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.009466 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.046494 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:15.095535 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.217139 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:14:15.239856 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.247959 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:15.321175 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:15.623869 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.648564 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.700964 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:15.798642 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:15.851137 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:16.099534 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:16.112390 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.134758 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.261523 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.394659 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.475215 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.506639 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:16.624200 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:16.643354 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:16.695925 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:14:16.722885 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:14:16.838377 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:16.949038 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.961781 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:16.967439 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:17.025651 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:17.090562 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.094079 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.144948 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.149000 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.204977 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:17.238453 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:17.303012 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:14:17.618175 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.667974 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.757750 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:17.873966 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:17.915839 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:17.957121 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:17.974855 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:18.102444 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.154958 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.235136 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:18.297807 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:18.380146 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.485107 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.486199 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:18.497665 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.505904 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:18.717567 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:18.718299 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:18.735492 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.923676 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:18.949836 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.061383 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:19.107091 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:19.160524 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.166759 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.226658 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:19.376838 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:19.493435 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:19.524256 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:19.569320 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.569427 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:19.650305 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:19.669306 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.693588 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:19.849346 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.919790 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.969931 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:19.980340 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:14:20.051544 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:20.296041 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:14:20.338658 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:20.482681 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:20.522949 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:20.560915 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:20.573367 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:20.926153 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:20.944078 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:20.945739 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:21.093129 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:21.099659 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:21.167948 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:21.169659 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:21.435909 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:21.570398 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:21.626252 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:21.692327 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:21.698865 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:21.804642 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:21.806644 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:21.811838 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:21.865816 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:21.892589 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:14:21.893196 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:22.007978 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:22.024413 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:22.060842 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:22.165725 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:22.232717 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:22.235657 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:22.276183 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:22.337850 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:22.471568 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:22.589504 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:22.803594 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:14:22.934125 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.026618 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:23.088474 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:23.101611 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:23.158561 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:23.193869 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.324677 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:23.523252 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:14:23.523933 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.553560 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.553642 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:23.659754 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:23.660512 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.670948 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:23.692283 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.704175 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:23.936013 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:24.078917 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:24.126698 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:24.146595 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:24.149925 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:24.200804 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:24.213812 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:24.265491 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:24.283447 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:24.488375 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:24.546649 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:24.634634 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:24.671325 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:24.709371 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:24.841112 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:24.845335 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:24.864768 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:25.121708 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:25.221966 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:25.361579 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:14:25.362177 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:14:25.495302 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:25.597852 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:25.685220 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:25.687841 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:25.723904 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:25.957413 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:26.048974 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.080230 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.120800 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:26.155328 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:26.209374 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:26.281854 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.364091 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.493693 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.502655 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.555084 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:26.770370 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:26.796844 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:26.844542 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:26.873893 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:26.903069 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:14:26.936242 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:26.960258 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:26.964408 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:26.974364 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:27.035009 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:27.105959 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:27.206576 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:27.209569 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:27.371047 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:27.444463 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:27.576085 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:27.595854 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:27.776258 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:27.870409 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:14:28.068910 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.144462 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:28.145066 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.240473 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.261296 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:28.302940 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:28.325724 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:28.423873 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.560974 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:28.597218 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.619985 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.650522 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:28.652364 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.795073 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:28.901961 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:28.953322 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:29.122573 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.162243 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:29.177773 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:29.202349 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:29.261746 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:29.319137 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:29.327919 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.359661 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:29.395776 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:29.416137 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.437875 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.454943 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.465060 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:29.504465 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.783921 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:29.875123 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:29.902798 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:30.028499 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:30.267535 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:30.380489 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:14:30.493129 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:30.493597 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:14:30.611547 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:30.691780 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:30.781985 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:30.807039 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:30.875311 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:30.939565 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.192797 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.219218 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.223256 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.374127 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:31.382565 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:31.546283 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.574100 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:31.620482 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:31.652683 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.655398 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.723367 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:31.734104 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:31.867689 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:31.906250 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:31.930176 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:31.956826 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:31.985018 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:32.030124 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:14:32.049160 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:32.103402 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:32.246517 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:32.248078 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:32.449002 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:32.602766 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:32.814119 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:14:32.845222 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:32.957943 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.038498 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:33.066457 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:33.128515 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.291292 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:33.292013 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.349193 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:33.370231 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.534330 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:33.538061 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.630080 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.701654 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:33.770130 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:33.884088 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:33.947283 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:34.060532 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:34.061065 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:34.065701 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 14:14:34.066319 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:34.171188 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.243741 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:34.263411 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:34.264108 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:34.351500 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:34.373112 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.380295 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.431335 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.453109 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:34.501186 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.544416 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.626647 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:34.640708 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:35.054899 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:35.209851 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:14:35.214725 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:35.314664 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:35.387430 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:14:35.419023 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:35.433598 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:35.596271 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:35.651202 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:35.658248 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:35.843061 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:36.018090 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.042096 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.206135 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:36.206688 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.248028 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:36.279845 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.285147 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:36.593611 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:36.609544 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.625366 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.727481 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:36.843104 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:36.888550 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:36.908161 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:36.921651 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:36.946229 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:36.950857 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:37.002465 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:37.026134 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:37.051496 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:37.075699 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:37.225983 12784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:37.251055 15288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:14:37.357154 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:37.407694 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:14:37.573051 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:37.755214 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:37.798796 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:37.841794 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.027006 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:38.053333 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:38.171783 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.261747 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.329023 12522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 14:14:38.342522 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:38.397300 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.413758 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.645741 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:38.746253 14893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 14:14:38.789161 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.836158 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:38.952710 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:38.997502 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.017915 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:39.116276 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.212219 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:39.223701 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:39.248586 15156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 14:14:39.259418 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:39.282503 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.310706 12784 consensus_peers.cc:485] T 727a51852ac2428eb23f7690fec56a16 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.311964 15288 consensus_peers.cc:485] T 3d6a7ed55367493e9edd0461d6da6313 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.399024 12915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:14:39.543365 12522 consensus_peers.cc:485] T 5b037b6095a644fda854cf0365b1a7df P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:39.560950 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.602244 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:39.616003 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:39.729180 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:39.805096 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:39.830924 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:14:39.894392 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:40.196007 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:40.225131 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:40.377911 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:40.468133 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:40.577153 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:40.596352 12653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:14:40.596932 12653 consensus_peers.cc:485] T 79e8de8cd4d2407c99ea4c5520d007f5 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:40.736987 15288 consensus_peers.cc:485] T 75e7c78545bf45be968aa742ac30603e P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:40.766642 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:40.777961 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:40.914471 12915 consensus_peers.cc:485] T d521005dcbf74e4fb9a67fe9992ce540 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:41.215147 15156 consensus_peers.cc:485] T c0bf116648874fc3a3ff54c7addf5e2d P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:41.222493 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
W20240501 14:14:41.235682 12915 consensus_peers.cc:485] T a5957da39df64253a2b8d7f388308719 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:41.236326 14893 consensus_peers.cc:485] T 0b37410addd44b5a9af05862e783590a P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:41.311816 14893 consensus_peers.cc:485] T 6f6e2acdbdc747a8a863d6e46d0d1b41 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:14:41.352281 3449 consensus_queue.cc:579] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.105s)
I20240501 14:14:41.362288 12585 consensus_queue.cc:237] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:14:41.364039 15352 raft_consensus.cc:1261] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:14:41.364107 12523 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer 4902d2609e6d44d089590a42e70356b2 (127.6.82.3:42215): Couldn't send request to peer 4902d2609e6d44d089590a42e70356b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c41acbf912c4dc0a68fe5b99df2f84b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.364622 3533 consensus_queue.cc:1035] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:14:41.365170 12522 consensus_peers.cc:485] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.366449 3542 raft_consensus.cc:2932] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:14:41.366865 15352 raft_consensus.cc:2932] T 0c41acbf912c4dc0a68fe5b99df2f84b P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } } }
I20240501 14:14:41.368083 12431 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c41acbf912c4dc0a68fe5b99df2f84b with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.370023 12443 catalog_manager.cc:5484] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index -1 to 2, NON_VOTER 4902d2609e6d44d089590a42e70356b2 (127.6.82.3) added. New cstate: current_term: 1 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:14:41.377782 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:14:41.528712 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:14:41.529546 3514 consensus_queue.cc:579] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.121s)
I20240501 14:14:41.548375 15353 consensus_queue.cc:237] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:14:41.549995 14958 raft_consensus.cc:1261] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:14:41.550550 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b49fa7fe41af439688ef801bb2044485. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.550561 3550 consensus_queue.cc:1035] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:14:41.550938 15288 consensus_peers.cc:485] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.552512 3395 raft_consensus.cc:2932] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.553337 14958 raft_consensus.cc:2932] T b49fa7fe41af439688ef801bb2044485 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.554837 12432 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b49fa7fe41af439688ef801bb2044485 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.556113 12442 catalog_manager.cc:5484] T b49fa7fe41af439688ef801bb2044485 P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.647819 3362 consensus_queue.cc:579] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.250s)
I20240501 14:14:41.654147 12717 consensus_queue.cc:237] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
W20240501 14:14:41.654996 12653 consensus_peers.cc:485] T f4010a078895473980e7108269ea7bc1 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:14:41.655891 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.655949 12849 raft_consensus.cc:1261] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:14:41.656356 12653 consensus_peers.cc:485] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.656397 3406 consensus_queue.cc:1035] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:14:41.658061 3362 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.658501 12849 raft_consensus.cc:2932] T e13e55ae52a949bc893de94082130157 P 4902d2609e6d44d089590a42e70356b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.659862 12429 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e13e55ae52a949bc893de94082130157 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.661347 12443 catalog_manager.cc:5484] T e13e55ae52a949bc893de94082130157 P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 6 to 7, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:14:41.669387 12915 consensus_peers.cc:485] T 68b04992f1f94b36b01dfde24a42318d P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:14:41.713656 3527 consensus_queue.cc:579] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.075s)
I20240501 14:14:41.720264 12849 consensus_queue.cc:237] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
W20240501 14:14:41.722025 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0e8e476bc46420d992a43bf3b4c73fd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.722518 12979 raft_consensus.cc:1261] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:14:41.722656 12784 consensus_peers.cc:485] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.722913 3461 consensus_queue.cc:1035] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:41.724534 3568 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.725016 12979 raft_consensus.cc:2932] T a0e8e476bc46420d992a43bf3b4c73fd P a6e94562bee149c2ace25672f0f9bf50 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.726161 12432 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a0e8e476bc46420d992a43bf3b4c73fd with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.728506 12442 catalog_manager.cc:5484] T a0e8e476bc46420d992a43bf3b4c73fd P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 7 to 8, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 2 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.733672 3550 consensus_queue.cc:579] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.305s)
W20240501 14:14:41.735524 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:14:41.737850 15353 consensus_queue.cc:237] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
W20240501 14:14:41.740059 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22e95701b67048ad871e7ff6fa4392cc. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.741546 15222 raft_consensus.cc:1261] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer f23f426391724d6688aafffe7ee017d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:14:41.741901 3550 consensus_queue.cc:1035] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:14:41.742046 15288 consensus_peers.cc:485] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.743383 3461 consensus_queue.cc:579] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.096s)
I20240501 14:14:41.743649 3554 raft_consensus.cc:2932] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.745210 12432 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 22e95701b67048ad871e7ff6fa4392cc with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.745590 15222 raft_consensus.cc:2932] T 22e95701b67048ad871e7ff6fa4392cc P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.747231 12849 consensus_queue.cc:237] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:14:41.747241 12442 catalog_manager.cc:5484] T 22e95701b67048ad871e7ff6fa4392cc P f23f426391724d6688aafffe7ee017d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "f23f426391724d6688aafffe7ee017d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.749148 12584 raft_consensus.cc:1261] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:14:41.749670 3461 consensus_queue.cc:1035] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:14:41.750102 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b744ec18c3474b11b72fbc2b160dbaef. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:41.750825 12784 consensus_peers.cc:485] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.752158 3461 raft_consensus.cc:2932] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.752656 12585 raft_consensus.cc:2932] T b744ec18c3474b11b72fbc2b160dbaef P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.754196 12432 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b744ec18c3474b11b72fbc2b160dbaef with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.756165 12442 catalog_manager.cc:5484] T b744ec18c3474b11b72fbc2b160dbaef P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index -1 to 2, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.756630 3449 consensus_queue.cc:579] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.014s)
I20240501 14:14:41.761880 12584 consensus_queue.cc:237] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:14:41.764297 12849 raft_consensus.cc:1261] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer 6a3cc0660eed427eaee38164a8c1d4c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:14:41.764710 3545 consensus_queue.cc:1035] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:14:41.764894 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 515d8f8e9d744aa4b2c8a4090c2f72e8. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:41.766505 12522 consensus_peers.cc:485] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.767683 3544 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.768146 12849 raft_consensus.cc:2932] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.768853 3572 consensus_queue.cc:579] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.492s)
I20240501 14:14:41.770030 12431 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 515d8f8e9d744aa4b2c8a4090c2f72e8 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.771505 12443 catalog_manager.cc:5484] T 515d8f8e9d744aa4b2c8a4090c2f72e8 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 7 to 8, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 2 leader_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:14:41.772394 12915 consensus_peers.cc:462] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:14:41.772759 3406 consensus_queue.cc:579] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.194s)
I20240501 14:14:41.778287 12979 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:14:41.778450 3565 consensus_queue.cc:579] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.101s)
W20240501 14:14:41.779670 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:14:41.780053 14957 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:41.780144 12718 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } }
I20240501 14:14:41.780519 3573 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:41.781224 12718 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:41.781643 3573 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:41.782140 14958 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:41.782584 3406 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:41.784579 3573 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:14:41.787703 12431 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25b5c40793b348bbb0e5251c8d553e30 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:41.787879 12718 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:14:41.789507 12979 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:14:41.789943 3406 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:14:41.790897 12718 consensus_queue.cc:237] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } }
I20240501 14:14:41.792590 12443 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.793023 12584 raft_consensus.cc:1261] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240501 14:14:41.793998 12654 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer f23f426391724d6688aafffe7ee017d8 (127.6.82.8:46801): Couldn't send request to peer f23f426391724d6688aafffe7ee017d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.794142 3406 consensus_queue.cc:1035] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:14:41.795656 3565 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
W20240501 14:14:41.797570 12653 consensus_peers.cc:485] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.797654 12429 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet abac52c109174021ac11360b0d22ccfa with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:14:41.798162 12979 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
W20240501 14:14:41.800069 12431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240501 14:14:41.800910 3565 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:14:41.801163 3594 ts_tablet_manager.cc:902] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: Initiating tablet copy from peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767)
I20240501 14:14:41.801293 12979 consensus_queue.cc:237] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:14:41.801713 3594 tablet_copy_client.cc:308] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.82.1:46767
I20240501 14:14:41.802485 12585 raft_consensus.cc:2932] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
W20240501 14:14:41.803675 12915 consensus_peers.cc:485] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b5c40793b348bbb0e5251c8d553e30. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.804174 12607 tablet_copy_service.cc:140] P 6a3cc0660eed427eaee38164a8c1d4c3: Received BeginTabletCopySession request for tablet 0c41acbf912c4dc0a68fe5b99df2f84b from peer 4902d2609e6d44d089590a42e70356b2 ({username='slave'} at 127.6.82.3:58555)
I20240501 14:14:41.804282 12607 tablet_copy_service.cc:161] P 6a3cc0660eed427eaee38164a8c1d4c3: Beginning new tablet copy session on tablet 0c41acbf912c4dc0a68fe5b99df2f84b from peer 4902d2609e6d44d089590a42e70356b2 at {username='slave'} at 127.6.82.3:58555: session id = 4902d2609e6d44d089590a42e70356b2-0c41acbf912c4dc0a68fe5b99df2f84b
I20240501 14:14:41.804430 14956 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:14:41.805331 12607 tablet_copy_source_session.cc:205] T 0c41acbf912c4dc0a68fe5b99df2f84b P 6a3cc0660eed427eaee38164a8c1d4c3: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:14:41.807227 14958 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:41.807873 3572 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:14:41.807940 12431 catalog_manager.cc:4573] TS ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 25b5c40793b348bbb0e5251c8d553e30: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111)
I20240501 14:14:41.809084 12443 catalog_manager.cc:5484] T 7130a0cb5bca433d8eb682e05b7a8f29 P 6a3cc0660eed427eaee38164a8c1d4c3 reported cstate change: config changed from index 5 to 6, NON_VOTER f23f426391724d6688aafffe7ee017d8 (127.6.82.8) added. New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: NON_VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: true } } }
I20240501 14:14:41.809568 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c41acbf912c4dc0a68fe5b99df2f84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:14:41.810029 14958 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:14:41.811105 12429 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7130a0cb5bca433d8eb682e05b7a8f29 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.812124 3594 tablet_copy_client.cc:783] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 0 data blocks...
I20240501 14:14:41.812299 3594 tablet_copy_client.cc:647] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:14:41.813110 12718 raft_consensus.cc:1261] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer a6e94562bee149c2ace25672f0f9bf50: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:41.813432 3594 tablet_copy_client.cc:523] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:14:41.813781 3572 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.814075 3329 consensus_queue.cc:1035] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:41.814512 12442 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e reported cstate change: config changed from index 7 to 8, NON_VOTER ab90e77f0027449bb3155980ca95593c (127.6.82.6) evicted. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } }
I20240501 14:14:41.815068 3594 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: Bootstrap starting.
I20240501 14:14:41.815493 3406 consensus_queue.cc:579] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.043s)
I20240501 14:14:41.816160 12431 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 25b5c40793b348bbb0e5251c8d553e30 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.817303 3360 consensus_queue.cc:579] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.252s)
I20240501 14:14:41.817842 14958 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.817350 12718 raft_consensus.cc:2932] T 25b5c40793b348bbb0e5251c8d553e30 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.819780 12717 consensus_queue.cc:237] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:14:41.821309 14956 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:41.821480 12979 raft_consensus.cc:1261] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:14:41.821689 3406 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:14:41.822053 12653 consensus_peers.cc:485] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abac52c109174021ac11360b0d22ccfa. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.822255 3601 consensus_queue.cc:1035] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:14:41.822873 12441 catalog_manager.cc:5484] T 25b5c40793b348bbb0e5251c8d553e30 P a6e94562bee149c2ace25672f0f9bf50 reported cstate change: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "a6e94562bee149c2ace25672f0f9bf50" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.823421 3565 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.824198 14956 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P 8a13c028ba8a4a659b74af6c265df95e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.824870 12979 raft_consensus.cc:2932] T abac52c109174021ac11360b0d22ccfa P a6e94562bee149c2ace25672f0f9bf50 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.825569 3568 consensus_queue.cc:579] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.523s)
I20240501 14:14:41.826367 12429 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet abac52c109174021ac11360b0d22ccfa with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.826681 12718 consensus_queue.cc:237] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:14:41.829650 12849 consensus_queue.cc:237] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:14:41.830909 12848 raft_consensus.cc:1261] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Refusing update from remote peer e6ab42bc38694f7f9fc8bfcc0a1483b7: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:14:41.831281 3362 consensus_queue.cc:1035] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:14:41.831849 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df57dee35f924801a23bd9b5c547a55f. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:41.832300 12653 consensus_peers.cc:485] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.832494 15353 raft_consensus.cc:1261] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:14:41.832587 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd5489eab5b04751bc8b3e5fb9e8cf73. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.832883 3578 consensus_queue.cc:1035] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:14:41.833016 3406 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
W20240501 14:14:41.833297 12784 consensus_peers.cc:485] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.835048 12848 raft_consensus.cc:2932] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.835340 12440 catalog_manager.cc:5484] T abac52c109174021ac11360b0d22ccfa P e6ab42bc38694f7f9fc8bfcc0a1483b7 reported cstate change: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.836886 12429 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df57dee35f924801a23bd9b5c547a55f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.841392 15222 consensus_queue.cc:237] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } }
I20240501 14:14:41.841652 12440 catalog_manager.cc:5484] T df57dee35f924801a23bd9b5c547a55f P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 8 to 9, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 2 leader_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.842128 3578 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.843745 12432 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd5489eab5b04751bc8b3e5fb9e8cf73 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 14:14:41.843909 15154 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2:40113): Couldn't send request to peer e6ab42bc38694f7f9fc8bfcc0a1483b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41a1a3342cae4af480f1e65164493219. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.844460 15353 raft_consensus.cc:2932] T bd5489eab5b04751bc8b3e5fb9e8cf73 P f23f426391724d6688aafffe7ee017d8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.844727 12585 raft_consensus.cc:1261] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:14:41.845322 3417 consensus_queue.cc:1035] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:14:41.846652 12440 catalog_manager.cc:5484] T bd5489eab5b04751bc8b3e5fb9e8cf73 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 9 to 10, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:14:41.847678 15156 consensus_peers.cc:485] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.850368 3594 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:14:41.850627 3594 tablet_bootstrap.cc:492] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: Bootstrap complete.
I20240501 14:14:41.850840 3594 ts_tablet_manager.cc:1367] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.004s
I20240501 14:14:41.851262 3594 raft_consensus.cc:348] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:14:41.851388 3594 raft_consensus.cc:729] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4902d2609e6d44d089590a42e70356b2, State: Initialized, Role: LEARNER
I20240501 14:14:41.851478 3594 consensus_queue.cc:260] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f23f426391724d6688aafffe7ee017d8" member_type: VOTER last_known_addr { host: "127.6.82.8" port: 46801 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: NON_VOTER last_known_addr { host: "127.6.82.3" port: 42215 } attrs { promote: true } }
I20240501 14:14:41.851734 3594 ts_tablet_manager.cc:1398] T 0c41acbf912c4dc0a68fe5b99df2f84b P 4902d2609e6d44d089590a42e70356b2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:14:41.852169 12607 tablet_copy_service.cc:342] P 6a3cc0660eed427eaee38164a8c1d4c3: Request end of tablet copy session 4902d2609e6d44d089590a42e70356b2-0c41acbf912c4dc0a68fe5b99df2f84b received from {username='slave'} at 127.6.82.3:58555
I20240501 14:14:41.852245 12607 tablet_copy_service.cc:433] P 6a3cc0660eed427eaee38164a8c1d4c3: ending tablet copy session 4902d2609e6d44d089590a42e70356b2-0c41acbf912c4dc0a68fe5b99df2f84b on tablet 0c41acbf912c4dc0a68fe5b99df2f84b with peer 4902d2609e6d44d089590a42e70356b2
I20240501 14:14:41.857676 3615 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:14:41.858213 12585 raft_consensus.cc:2932] T 41a1a3342cae4af480f1e65164493219 P 6a3cc0660eed427eaee38164a8c1d4c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } } }
I20240501 14:14:41.859134 12429 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 41a1a3342cae4af480f1e65164493219 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.861554 12442 catalog_manager.cc:5484] T 41a1a3342cae4af480f1e65164493219 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 4 to 5, NON_VOTER e6ab42bc38694f7f9fc8bfcc0a1483b7 (127.6.82.2) added. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: NON_VOTER last_known_addr { host: "127.6.82.2" port: 40113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.880620 3525 consensus_queue.cc:579] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.283s)
I20240501 14:14:41.892077 14956 consensus_queue.cc:237] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } }
I20240501 14:14:41.893683 15221 raft_consensus.cc:1261] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Refusing update from remote peer 8a13c028ba8a4a659b74af6c265df95e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:14:41.894292 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.894351 3625 consensus_queue.cc:1035] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 14:14:41.898902 14893 consensus_peers.cc:485] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e -> Peer 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1:46767): Couldn't send request to peer 6a3cc0660eed427eaee38164a8c1d4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cee040fd7ece4b8d8af328f9205dae20. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.911772 3625 raft_consensus.cc:2932] T cee040fd7ece4b8d8af328f9205dae20 P 8a13c028ba8a4a659b74af6c265df95e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.913543 12430 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cee040fd7ece4b8d8af328f9205dae20 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.913797 15221 raft_consensus.cc:2932] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.918435 12441 catalog_manager.cc:5484] T cee040fd7ece4b8d8af328f9205dae20 P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a3cc0660eed427eaee38164a8c1d4c3 (127.6.82.1) added. New cstate: current_term: 1 leader_uuid: "8a13c028ba8a4a659b74af6c265df95e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: VOTER last_known_addr { host: "127.6.82.5" port: 33177 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } } peers { permanent_uuid: "6a3cc0660eed427eaee38164a8c1d4c3" member_type: NON_VOTER last_known_addr { host: "127.6.82.1" port: 46767 } attrs { promote: true } } }
I20240501 14:14:41.955991 3616 consensus_queue.cc:579] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.461s)
W20240501 14:14:41.958117 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 14:14:41.961040 15221 consensus_queue.cc:237] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } }
I20240501 14:14:41.963141 12718 raft_consensus.cc:1261] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 66e2dc34330848c3b8c2dd15898d34c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:14:41.965031 3616 consensus_queue.cc:1035] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:14:41.966285 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4:43151): Couldn't send request to peer a6e94562bee149c2ace25672f0f9bf50. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:41.966387 15156 consensus_peers.cc:485] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.972740 3417 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.974318 12429 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e8226993467944dcb24d1efcda07538a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.974810 3527 consensus_queue.cc:579] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Leader has been unable to successfully communicate with peer ab90e77f0027449bb3155980ca95593c for more than 300 seconds (300.313s)
I20240501 14:14:41.976912 12718 raft_consensus.cc:2932] T e8226993467944dcb24d1efcda07538a P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } } }
I20240501 14:14:41.977284 12442 catalog_manager.cc:5484] T e8226993467944dcb24d1efcda07538a P 66e2dc34330848c3b8c2dd15898d34c8 reported cstate change: config changed from index 4 to 5, NON_VOTER a6e94562bee149c2ace25672f0f9bf50 (127.6.82.4) added. New cstate: current_term: 1 leader_uuid: "66e2dc34330848c3b8c2dd15898d34c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "66e2dc34330848c3b8c2dd15898d34c8" member_type: VOTER last_known_addr { host: "127.6.82.7" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e94562bee149c2ace25672f0f9bf50" member_type: NON_VOTER last_known_addr { host: "127.6.82.4" port: 43151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:14:41.980361 12849 consensus_queue.cc:237] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } }
W20240501 14:14:41.982322 12783 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5:33177): Couldn't send request to peer 8a13c028ba8a4a659b74af6c265df95e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 218f8a4fae324dc7bc535e6a18addb61. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:14:41.984007 12784 consensus_peers.cc:485] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 -> Peer ab90e77f0027449bb3155980ca95593c (127.6.82.6:37673): Couldn't send request to peer ab90e77f0027449bb3155980ca95593c. Status: Network error: Client connection negotiation failed: client connection to 127.6.82.6:37673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:14:41.985625 12718 raft_consensus.cc:1261] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Refusing update from remote peer 4902d2609e6d44d089590a42e70356b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:14:41.986145 3568 consensus_queue.cc:1035] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:14:41.988202 3568 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:14:41.988613 12718 raft_consensus.cc:2932] T 218f8a4fae324dc7bc535e6a18addb61 P e6ab42bc38694f7f9fc8bfcc0a1483b7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2" port: 40113 } } peers { permanent_uuid: "ab90e77f0027449bb3155980ca95593c" member_type: VOTER last_known_addr { host: "127.6.82.6" port: 37673 } attrs { promote: false } } peers { permanent_uuid: "8a13c028ba8a4a659b74af6c265df95e" member_type: NON_VOTER last_known_addr { host: "127.6.82.5" port: 33177 } attrs { promote: true } } }
I20240501 14:14:41.989846 12432 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 218f8a4fae324dc7bc535e6a18addb61 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:14:41.993305 12442 catalog_manager.cc:5484] T 218f8a4fae324dc7bc535e6a18addb61 P 4902d2609e6d44d089590a42e70356b2 reported cstate change: config changed from index 6 to 7, NON_VOTER 8a13c028ba8a4a659b74af6c265df95e (127.6.82.5) added. New cstate: current_term: 1 leader_uuid: "4902d2609e6d44d089590a42e70356b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4902d2609e6d44d089590a42e70356b2" member_type: VOTER last_known_addr { host: "127.6.82.3" port: 42215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ab42bc38694f7f9fc8bfcc0a1483b7" member_type: VOTER last_known_addr { host: "127.6.82.2"